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^rgor^ AND APPARATUS FOR MEASURING ORIENTATION AND 

DISTANCE 

Cross Reference to Relat ed Applications 

The present application claims the benefit, under 35 U.S.C. §1 19(e), of U.S. 
Provisional Application Serial No. 60/164,754, entitled "Image Metrology System," and of 
U.S. Provisional Application Serial No. 60/212,434, entitled "Method for Locating 
Landmarks by Machine Vision," which applications are hereby incorporated herein by 
reference. 

Field of the Invention 

The present invention relates to various methods and apparatus for facilitating 
measurements of orientation and distance, and more particularly, to orientation and distance 
Measurements for image metrology applications. 

Description of the Related Art 

A, Introduction 

Photogrammetry is a technique for obtaining information about the position, size, and 
shape of an object by measuring images of the object, instead of by measuring the object 
directly. In particular, conventional photogrammetry techniques primarily involve 
determining relative physical locations and sizes of objects in a three-dimensional scene of 
interest from two-dimensional images of the scene (e.g., multiple photographs of the scene). 

In some conventional photogrammetry applications, one or more recording devices 
(e.g., cameras) are positioned at different locations relative to the scene of interest to obtain 
multiple images of the scene from different viewing angles. In these applications, multiple 
images of the scene need not be taken simultaneously, nor by the same recording device; 
however, generally it is necessary to have a number of features in the scene of interest appear 
in each of the multiple images obtained from different viewing angles. 



In conventional programmetry, knowledge of the spatial relationship between the 
scene of interest and a given recording device at a particular location is required to determine 
information about objects in a scene from multiple images of the scene. Accordingly, 
conventional photogrammetry techniques typically involve a determination of a position and 
an orientation of a recording device relative to the scene at the time an image is obtained by 
the recording device. Generally, the position and the orientation of a given recording devi6e 
relative to the scene is referred to in photogrammetry as the "exterior orientation" of the 
recording device. Additionally, some information typically must be known (or at least 
reasonably estimated) about the recording device itself (e.g., focussing and/or other 
calibration parameters); this information generally is referred to as the "interior orientation" of 
the recording device. One of the aims of conventional photogrammetry is to transform two- 
dimensional measurements of particular features that appear in multiple images of the scene 
into actual three-dimensional information (i.e., position and size) about the features in the 
scene, based on the interior orientation and the exterior orientation of the recording device 
used to obtain each respective image of the scene. 

In view of the foregoing, it should be appreciated that conventional photogrammetry 
techniques typically involve a number of mathematical transformations that are applied to 
features of interest identified in images of a scene to obtain actual position and size 
information in the scene. Fundamental concepts related to the science of photogrammetry are 
described in several texts, including the text entitled Close Range Photogrammetry and 
Machine Vision, edited by K.B. Atkinson, and published in 1996 by Whittles Publishing, 
ISBN 1-870325-46-X, which text is hereby incorporated herein by reference (and hereinafter 
referred to as the "Atkinson text"). In particular, Chapter 2 of the Atkinson text presents a 
theoretical basis and some exemplary fundamental mathematics for photogrammetry. A 
summary of some of the concepts presented in Chapter 2 of the Atkinson text that are 
germane to the present disclosure is given below. The reader is encouraged to consult the 
Atkinson text and/or other suitable texts for a more detailed treatment of this subject matter. 



Additionally, some of the mathematical transformations discussed below are presented in 
greater detail in Section L of the Detailed Description, as they pertain more specifically to 
various concepts relating to the present invention. 

B. The Central Perspective Projection Model 

Fig. 1 is a diagram which illustrates the concept of a "central perspective projection," 
which is the starting point for building an exemplary functional model for photogrammetry. 
In the central perspective projection model, a recording device used to obtain an image of a 
scene of interest is idealized as a "pinhole" camera (i.e., a simple aperture). For purposes of 
this disclosure, the term "camera" is used generally to describe a generic recording device for 
acquiring an image of a scene, whether the recording device be an idealized pinhole camera or 
various types of actual recording devices suitable for use in photogrammetry applications, as 

discussed further below. 

In Fig. 1, a three-dimensional scene of interest is represented by a reference coordinate 
system 74 having a reference origin 56 (O r ) and three orthogonal axes 50, 52, and 54 (x n y n 
and z r , respectively). The origin, scale, and orientation of the reference coordinate system 74 
can be arbitrarily defined, and may be related to one or more features of interest in the scene, 
as discussed further below. Similarly, a camera used to obtain an image of the scene is 
represented by a camera coordinate system 76 having a camera origin 66 (O c ) and three 
orthogonal axes 60, 62, and 64 (x c , y c , and z c , respectively). 

In the central perspective projection model of Fig. 1, the camera origin 66 represents a 
pinhole through which all rays intersect, passing into the camera and onto an image 
(projection) plane 24. For example, as shown in Fig. 1, an object point 51 (A) in the scene of 
interest is projected onto the image plane 24 of the camera as an image point 51' (a) by a 
straight line 80 which passes through the camera origin 66. Again, it is to be appreciated that 
the pinhole camera is an idealized representation of an image recording device, and that in 



practice the cameraorigin 66 may represent a "nodal point" of a lens or .ens system of an 
actual camera or other recording device, as discussed further below. 

,n the model of Fig. 1, the camera coordinate system 76 is oriented such that the z„ 
axis 64 defines an optical axis 82 of the camera. Ideally, the optical axis 82 is orthogonal to 
the image plane 24 of the camera and intersect the image plane at an image plane ongm 67 
(P» Accordingly, the image plane 24 generally is defined by two orthogonal axis x, and,,, 
which respectively are parallel to the * axis 60 and the y. axis 62 of the camera coordrnate 
system 76 (wherein the * axis 64 of the camera coordinate system 76 is directed away from 
the image plane 24). A distance 84 (4 between the camera origin 66 and the image plane 
origin 67 typically is referred to as a "principal disunce" of the camera. Henc, in terms of 
the camera coordinate system 76, the image plane 24 is located at z, = -d. 

In Fig. 1, the object points and image point a each may be described in terms of the* 
three-dimensional coordinates in the camera coordinate system 76. For purposes of the 
present disclosure, the notation 



Pb 



is introduced generally to indicate a set of coordinates for a point B in a coordinate system S. 
Likewise, it should be appreciated that this notation can be used to express a vector from the 
origin of the coordinate system Sto the point B. Using the above notation, individual 
coordinates ofthe set are identified by '/».«. 'P. 00, and «/>.«. for example. Additionally, 
it should be understood that the above notation may be used to describe a coordinate system S 
having any number of (e.g., two or three) dimensions. 

With the foregoing notation in mind, the set of three y-, and .-coordinates for the 
object point A in the camera coordinate system 76 (as well as the vector O c A from the camera 
origin 66 to the object point A) can be expressed as <P A . Similarly, the set of three 
coordinates for the image point a in the camera coordinate system (as well as the vector O c a 
from the camera origin 66 to the image point a) can be expressed as <P a , wherein the z- 
coordinate of c P a is given by the principal distance 84 (i.e., c P a (z) = -d ). 



From the projective model of Fig. l,i, may be appreciated that the vectors 'P A and 
; are opposite in direction and proportional in iength. In particular, the following ratios may 
beUtten for the coordinates of the object point A and 4c image point a in the camera 
coordinate system: 

c P a {z) C P A & 

and 

•P.iy) = 'W> 
'/>« 'f/W 

By rearranging the above equation and making the substitution'^) - -0 for the principal 
distance 84, the x- and v-coordinates of the image point a in the camera coordinate system 
may be expressed as: 



(1) 



and 



(2) 



„ should be appreciated ma, since the respective „ and y axes of the camera coordinate system 
76 and the image plane 24 are parallel, Eqs. (1) and (2) also represent the image coordtnates 
(sometimes referred to as "photo-coordinates") of the image point a in the image plane 24. 
Accordingly, the *- ^coordinates of the image point a given by Eos. (1) and (2) aiso may 
be expressed respectively as 'P. (*) and 'P. W, where the left superscript , represents the mo- 
dimensional image coordinate system given by the * axis and the* axis in the image plane 
24. 

From Eqs. (1) and (2) above, it can be seen that by kno^ng the principal distance d 
and the coordinates of the object point A in the camera coordinate system, the image 
coordinates T a (v) of the image point « may be uniquely determined. However,, 

should also be appreciated that if the principal distance d and the image coordmates < P. (*) 



^ M of the image point a are known, the three-dimensional -ruinates of the obtect 
^owns in two equation, For this reason, — photogrammetry technics 

Irrletnetnree-— .coordinate S of .e o.ec, point in .e S cen, ^ — 
imag e re,— is discussed further below in the Section O of the Descnpnon of 
Related Art, entitled "Intersection." 

C Coordinate System Transformations 

While E,,<1) and (2) relate the image points the object points in Fig. 1 m terms 
„ f the camera coordinate system 76, one of the aims of conventionai photogrammetry 

.heir three-dimensionai coordinates in a reference coordinate system for the scene (e.g the 
« coordinate system 74 shown in Fig. !)• Accordingly, one important aspect of 

relationship (i.e., reiative position and orientation, of me camera coordinate system 6fo a 

^-.p^^-^^ - ^ 74, "" ta,mm ^: of . 

^relationship commonly isrefcrredtoinphotogrammetryasthe — onen^on ofa 
camera, and is referred to as such throughout mis disclosure. 

Fig 2 is a diagram illustrating some fundamental concepts related to coordmate 

side of Fig. 2) to the camera coordinate system 76 (shown on the left side of Frg. 2). The 

Detailed Description. . 

In Fig . 2, the object point 5 1 (,) may be described in terms of its tbree-drmenstonal 
coordinates in either the reference coordinate system 74 or the camera coordinate system 76. 



,„ particular, using the notation introduced above, the coordinates of ft. point A tn the 
reference coordinate system 74 (as we,, as a firs, vector 77 from the origin 56 of the reference 
coordinate system 74 to the points) can be expressed as 'P. . Simiiarly, as discussed above, 
,he coordinates of the point A in the camera coordinate system 76 (as well as a second vector 
79 from the origin 66 of the camera coordinate system 76 to the object points) can be 
expressed as -? t , wherein the left superscripts , and c represent the reference and camera 

coordinate systems, respectively. 

Also indicated in Fig. 2 is a third "translation" vector 78 from the origin 56 of the 
reference coordinate system 74 to the origin 66 of the camera coordinate system 76. T*e 
translation vector 78 may be expressed in the above notation as ' />„. . In particular, the vector 
' P designates the location (i.e., position) of the camera coordinate system 76 with respect ,0 
the reference coordinate system 74. Stated alternatively, the notation > P„, represents an x- 
coordinate, a ^coordinate, and a z-coordinate of the origin 66 of the camera coordinate 
system 76 with respect to the reference coordinate system 74. 

In addition to a translation of one coordinate system to another (as indicated by the 
vector 78) Fig. 2 illustrates that one of the reference and camera coordinate systems may be 
rotated in three-dimensiona! space with respect to the other. For example, an orientation of 
the camera coordinate system 76 with respect to the reference coordinate system 74 may be 
defined by a rotation about any one or more of the *, y, and z axes of one of the coordinate 
systems For purposes of the present disclosure, a rotation of, degrees about an x axts ts 
referred to as a "pitch" rotation, a rotation of « degrees about *y axis is referred to as a "yaw- 
rotation, and a rotation of/! degrees about a , axis is referred to as a "roll" rotation. 

With this terminology in mind, as shown in Fig. 2, a pitch rotation 68 of the reference 
coordinate system 74 about the * axis 50 ahers the position of the* axis 52 and the „ axts 54 
so tha, they respectively may be parallel aligned with the* axis 62 and the * axis 64 of the 
camera coordinate system 76. Similarly, a yaw rotation 70 of the reference coordinate system 
about the y, axis 52 alters the position of the * axis 50 and the z, axis 54 so mat they 
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respective* may be parallel aligned with the X , axis 60 and ft. „ axis 64 of the eamera 
coordinate system. Likewise, a rolL rotation 72 of the reference coordinate system about the * 
axis 54 alters the position of the * axis 50 and the y, axis 52 so tha, they respectively may be 
parallel aligned with the * axis 60 and the* axis 62 of the camera coordinate system. It 
should be appreciated mat, conversely, the camera coordinate system 76 may be rotated about 
. one or more of its axes so that its axes are parallel aligned with the axes of the reference 
coordinate system 74. 

,„ sum, an orientation of the camera coordinate system 76 with respect to the reference 
coordinate system 74 may be given in terms of three rotation angles; namely, a pitch rotation 
angle fr), a yaw rotation angle («), and a roll rotation angle («. This orientation may be 
expressed by athree-by-three rotation matrix, wherein each of the nine rotation matnx 
elements represents a trigonometric function of one or more of the yaw, roll, and pitch angles 
a, ft and respectively. For purposes of the present disclosure, the notation 



52 n 



is used to represent one or more rotation matrices that implement a rotation from the 
coordinate system S! to the coordinate system S2. Using this notation, ;* denotes a rotation 
from the reference coordinate system to tire camera coordinate system, and ;* denotes .he 
inverse rotation (i.e., a rotation from the camera coordinate system to the reference coordmate 
system). It should be appreciated mat since these rotation matrices are orthogonal, the mverse 
ofagivenrotationmatrixisequivalenttoitstranspose; accordingly, • ^should 

also be appreciated mat rotations between the camera and reference coordinate systems shown 
in Fig 2 implicitly include a 180 degree yaw rotation of one of the coordinate systems about 

i, sy axis, so that the respective , axes of the coordinate systems are opposite in sense (see 

Section L of the Detailed Description). 

By combining the concepts of translation and rotation discussed above, the coordinates 

of the object point A in the camera coordinate system 76 shovm in Fig. 2, based on the 

coordinates of the point A in the reference coordinate system 74 and a transformation (,.e., 
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Elation and rotation) from the reference coordinate system to the camera coordinate 
system, are given by the vector expression: 

<p A . -R T, + 'Pa, ■ (3) 

Likewise, me coordinates of the point A in me reference coordinate system 74, based on the 

coordinates of ft. point A in the camera coordinate system and a transformation (,e., . 

Elation and rotation) from the camera coordinate system to Are reference coordmate 

system, are given by the vector expression: 

T, = ;« ^ + A ■ (4) 

where -j. and where for the transiation vector 78, 'P„, 'V EachofEqs.<3) 

and ^includes six parameters which constitute the exterior orientation of the camera; 
namely, tee position parameters in the respective translation vectors ' P 0 , and ' P a , Ccthe 
respective *, v, and .-coordinates of one coordinate system origin in terms of the other 
coordinate system), and three orientation parameters in the respective rotation matnces , R 
and ;R (i.e., the yaw, roll, and pitch rotation angles a, fi, and y). 

Eqs. (3) and (4) alternatively may be written using the notation 



which is introduced to generally represent a coordinate transformation function of the 
argument in parentheses. The argument in parentheses is a se, of coordinates mure 
coordinate system SI, and the transformation function T transforms these coordinates to 
coordinates in the coordinate system S2. In general, it should be appreciated that the 
transformation func.cn rmay be a linear or a nonlinear function; in particular, the coordmate 
systems SI and S2 may or may not have the same dimensions, .n the following discuss™, 
the notation 7"' is used herein to indicate an inverse coordinate transformahon (e.g., 
«r -(•)- •) . the argument in parenthesis is a set of coordinates in the 

coordinate system S2). 

Using the notation of Eq. (5), Eqs. (3) and (4) respectively may be rewritten as 
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°p A = VTCP A ), (6) 



and 

(7) 



r P A = r c T( c P A ). 



wherein the transformation functions \T and ? represent mappings between the three- 
dimensional reference and camera coordinate systems, and wherein 7 = 
transformations are inverses of each other). Each of the transformation functions^ and >J 
includes a rotation and a translation and, hence, the six parameters of the camera exterior 
orientation. 

With reference again to Fig. 1, it should be appreciated that the concepts of coordinate 
system transformation illustrated in Fig. 2 and the concepts of the idealized central 
perspective projection model illustrated in Fig. 1 may be combined to derive spatial 
transformations between the object point 51 (A) in the reference coordinate system 74 for the 
scene and the image point 51' (a) in the image plane 24 of the camera. For example, known 
coordinates of the object point A in the reference coordinate system may be first transformed 
using Eq. (6) (or Eq. (3)) into coordinates of the point A in the camera coordinate system. The 
transformed coordinates may be then substituted into Eq, (1) and (2) to obtain coordinates of 
the image point a in the image plane 24. In particular, Eq. (6) may be rewritten in terms of 
each of the coordinates of <P A , and the resulting equations for the respective coordinates 
C P A (*), C P A 00> ^d ' P A (z) may be substituted into Eq, (1) and (2) to give two "collinearity 
equations'^, for example, the Atkinson text, Ch. 2.2), which respectively relate the x- and 
,-image coordinates of the image point a directly to the three-dimensional coordinates of the 
object point A in the reference coordinate system 74. It should be appreciated that one object 
point A in the scene generates two such collinearity equations (i.e., one equation for each x- 
and >>-ima g e coordinate of the corresponding image point a), and that each of the collinearity 
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well as terms related to the six 



equations includes the principal distance d of the camera, 
exterior orientation parameters (i.e., three position and three orientation parameters) of the 



camera. 



D Determining Exterior Orientation Parameters: "Resection" 
,f the exterior orientation of a given camera is no. known a prior, (which is often the 
case in many photograrnrnetry applications), one important aspect of conventional 
programmetry technics involves determining the parameters of the camera extenor 
orientation for each different image of the scene. The evaluation of the six parameters of the 
camera exterior orientation from a single image of the scene commonly is referred to m 
photograrnrnetry as "resection." Various convention* resection methods are known, wtth 
different degrees of complexity in the methods and accuracy in the determination of the 

exterior orientation parameters. 

,„ conventional resection methods, generally the principal distance d of the camera rs 
known or reasonabiy estimated a priori (see Eqs. (!) and (2)). Additionally, at leas, three 
nonlinear "control points" are selected in the scene of interest that each appear in an .mage 
of the scene. Control points refer to features in tire scene for which actual relative position 
and/or size information in tire scene is known. Specifically, tire spatial relationship between 
tire control points in the scene must be known or determined (e.g., measured) aprior, such 
that the three-dimensional coordinates of each control point are known in the reference 
coordinate system. In some instances, a. leas, three non-collinear control points are 
particularly chosen ,o actually define tire reference coordinate system for tire scene. 

As discussed above in Section B of the Description of the Related Art, conventional 
photograrnrnetry techniques typically require multiple images of a scene to determine 
unknown three-dimensiona, position and s to information of objects of interest in the scene: 
According in many instances, the control points for resection need to he carefully selected 
such that they are visible in multiple images which are respectively obtained by cameras a, 
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different — , so tha, the exterior orientation of each camera may be determined wtth 
respect to the same control points (i.e., a common reference coordinate system). Often, 
meeting such contro, points is no, a triviai task; for example, it may be necessary to plan a 
photo-survey of the scene of Interest to insure that not only are a sufficient number 
points avai,ab,e in the scene, bu, that candidate contro, points are no, obscured a, different 
camera locations by Cher features in me scene. Additional.,, in some instances, i, may be 
incumbent on a photogrammetry analyst to identic the same control points in muUiple images 
accurately (i.e., "matching" of corresponding images of control points) to avoid errors m the 
determination of the exterior orientation of cameras a, different iocations with respect to a 
common reference coordinate system. These and other issues reiated to corresponding pomt 
identification in multiple images are discussed further below in the Sections G and H of the 
Description of the Related Art, entitled " and T - 

Bundle Adjustments," respectively. 

I„ conventional resection methods, each control point corresponds to two collineanty 
conations which respectively relate the*- and,-image coordma.es of a contro. point as ,t 
appears in „ image to the three-dimensiona, coordina.es of the contro! point in .he reference 
coordinate system 74 (as discussed above in Section C of the Description of the Related Art). 
For each contro, point, the respective image coordinates in the two cohinearity equations are 
obtained from tire image. Additionally, as discussed above, the principal distance of the 
camera generally is known or reasonably estimated a prior,, and the reference system 
coordinates of each contro! point are known a priori (by definition). Accordingly, each 
coUinearity equation based on me idealized oinho.e camera mode, of Fig. 1 (i.e., using Eqs. 
(1) and (2)) has only six unknot parameters (i.e., three position and three orientation 
parameters) corresponding to the exterior orientation of me camera. 

In view of the foregoing, using at leas, three control points, a system of at leas. s« 
collinearity equations (two for each control point) in six unknowns is generated. In some 
conventional resection methods, only «hree non-collinear contro, points are used to dtrectly 
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s „,ve (i.e., without using any approximate initial values for the unknown parameters) such a 
system of six eu.ua.ions in six unknowns to give an estimation of the exterior orientation 
parameters. In other conventional resection methods, a more rigorous iterative leas, squares 
estimation process is used to soive a system of a. leas, six collinearity equations. 

In an iterative estimation process for resection, often more than three control points are 
used to generate more titan six equations to improve the accuracy of the estimation. 
Additionally, in such iterative processes, approximate values for the exterior orientation 
parameters that are sufficiently close to the final values typically must be known a prior, (e.g., 
using direct evaluation) for the iterative process to converge; hence, iterative resection 
methods typicauy involve «o steps, namely, initial estimation followed by an iterative leas, 
squares process. The accuracy of me exterior orientation parameters obtained by such 
iterative processes may depend, in part, on the number of control points used and me spatial 
distribution of the control points in the scene of interest; generally, a greater number of well- 
distributed control points in me scene improves accuracy. Of course, it should be appreciated 
mat me accuracy with which me exterior orientation parameters are determined in turn affects 
me accuracy with which position and size information about objects in the scene may be 
determined from images of the scene. 

E Camera Modeling: Interior Orientation and Distortion Effects 
The accuracy of the exterior orientation parameters obtained by a given resection 
method also may depend, a, leas, in pari, on how accurate* the camera itself is modeled. For 
example, while Fig. 1 illustta.es an ideahzed projection model (using a pinhole camera) that ts 
described by E,s. (1) and (2), in practice an actual camera tha. includes various focussmg 
element, (e.g., a lens or a lens system) may affec. .he projection of an objec, poin, onto an 
image plane of me recording device in a manner ma, deviates from me idealized model of Ftg. 
1 In particular, Eqs. (1) and (2) may in some cases need to be modified ,o include other 
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terms «ha. take into consideration the effects of various structural dements of the camera, 
depending on the degree of accuracy desired in a particular photogrammetry apphcatton. 

Suitable recording devices for programmetry applications generally may be 
separated into three categories; namely, film cameras, video cameras, and digital devices (e.g., 
digital cameras and scamrers). As discussed above, for purposes of toe present disclosure, the 
term "camera" is used herein generically .o describe any one of various recording devrces for 
acquiring an image of a scene that is suitable for use in a given photogrammetry application. 
Some cameras are designed specifically for photogrammetry applications (e.g., "metric" 
cameras), while outers may be adapted and/or calibrated for particular photogrammetry use, 

A camera may employ one or more focussing elements that may be essentially fixed to 
implement a particular focus setting, or mat may be adjustable to implement a number of 
different focus settings. A camera with a lens or lens system may differ from the idealized 
pinhole camera of the central perspective projection model of Fig. 1 in that the princrpal 
distonce 84 beWeen the camera origin 66 (i.e., the nodal point of the lens or lens system) may 
change with lens focus setting. Additionally, unlike the idealized model shown in Fig. 1, the 
optical axis 82 of a camera with a lens or lens system may not intersect the image plane 24 
precisely a, the image plane origin O, , but rather at some point in the image plane mat is offset 
from the origin O,. For purposes of this disclosure, the point at which the optical axrs 82 
actuaHy intersects the image plane 24 is referred to as the "principal point" in the image plane. 
The respective x- andy-coordinates in the image plane 24 of the principal point, together vMh 
the principal distance for a particular focus setting, commonly are referred to in 
photognuumetry as ••interior orientation" parameters of the camera, and are referred to as such 

throughout this disclosure. 

Traditionally, metric cameras manufactured specifically for photogrammetry 
applications are designed to include certain features that ensure close conformance to the 
central perspective projection model of Fig. 1 . Manufacturers of metric cameras typically 
provide calibration information for each camera, including coordinates for the principal point 



# 
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in the image plane 24 and calibrated principal distances 84 corresponding to specific *»1 
settings (i.e., the interior orientation parameters of &e camera for different focal settings). 
These three interior orientation parameters may be used to modify E,s. (!) and (2) so as to 
more accurately represent a model of the camera. 

Film cameras record images on photographic film. Film cameras may be 
manured specifically for programmetry applications (i.e., a metric film camera), for 
example, by including "fiducial marks" (e.g., the points f„ 6, f 3 , and f, shown in Fig. 1) mat 
are fixed to the camera body to define the x, and y, axes of the image plane 24. Alternative*, 
for example, some conventional (i.e., non-metric) Elm cameras may be adapted to include 
fihn-type inserts that a«ach to the film rails of the device, or a glass plate mat is fixed in the 
camera body a, the image plane, on which fiducial marks are printed so as to provide for an 
image coordinate system for photogrammetry applications. In some cases, film forma, edges 
may be used to define a reference for the image coordinate system. Various degrees of 
accuracy may be achieved with the foregoing examples of film cameras for photogrammetry 
applications. With non-metric film cameras adapted for photogrammetry applications, 
typically the interior orientation parameters must be determined through calibration, as 

discussed further below. 

Digital cameras generally employ a two-dimensional array of light sensitive elements, 
or "pixels" (e.g„ CCD image sensors) disposed in the image plane of the camera. The rows 
and cotamns of pixels typically are used as a reference for the » and y, axes of the image 
p,ane 24 shown in Fig. 1, thereby obviating fiducial marks as often used with metnc film 
cameras. Generally, both digital cameras and video cameras employ CCD arrays. However, 
images obtained using digital cameras are stored in digital format (e.g., in memory or on 
disks), whereas images obtained using video cameras typically are stored in analog format 

(e.g., on tapes or video disks). 

images stored in digital format are particularly useful for photogrammetry applications 
implemented using computer processing techniques. Accordingly, images obtained using a 
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video camera may be placed into digita. forma, using a variety of commercial availabie 
converters (e.g., a "frame grabber" and/or digitizer board,. Similarly, images .Ren usmg a 
mm camera may be placed into digital format using a digita, scanner which, like a dr g ,ta, 
camera, generally employs a CCD pixel array. 

Digital image recording devices such as digital cameras and scanners introduce 
another parameter of interior orientation; namely, an aspect ratio (i.e., a digitizing scale, or 
ra ,io of pixel density aiong the * axis to pixel den* along the v, axis) of the CCD array in 
the image plane. Accordingly, a to., of four parameters; name.y, principal distance, aspect 
ratio and respective *- and v-coordina.es in the image platte of the principal poin, typically 
constitute the interior orientation of a digital recording device. If an image is taken usmg a 
mm camera and converted to digital forma, using a scaler, these four parameters of intenor 
orientation may apply ,0 the combination of the film camera and the scanner viewed 
hypothetical* as a single image recording device. As with metric Sim cameras, 
^ufacturers of some digital image recording devices may provide calibration informal 
for each device, inciuding the four interior orientation parameters. With outer digital devices, 
however, these parameters may have to be determined through calibration. As discussed 
above, the four interior orientation parameters for digital devices may be used to modrfy Eos. 
(1) and (2) so as to more accurately represent a camera model. 

,„ mm cameras, video cameras, and digita. image recording devices such as d.grtal 
^eras and scamrers, other characteristics of focussing elements may contribute to a 
deviation from the idealized central perspective projection mode, of Fig. 1. Fo, example, 
•,adial distortion" of a lens or lens system refers ,0 nonlinear variations in angular 
m agni„ca,ion as a function of angle of incidence of an optical ray to the lens or tens system. 
Radial distortion can introduce differential errors to the coordinates of an image pom, as a 
action of a radial distance of me image poin, from me principa! point in the image plane, 
according to the expression 

SR = K l R 3 +K 2 R 5 +K,R 1 , W 
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where * is «he radial distance of the image point from the principal point, and the coeffic.ents 
K, K 2 and K 3 are parameters that depend on a particular focal setting of the lens or lens 
system (see, for example, the Atkinson text, Ch. 2.2.2). Other models for radial distortion are 
sometimes used based on different numbers of nonlinear terms and orders of power of the 
terms (e g., R\ R\ In any case, various mathematica! models for radial distortion typically 
include two to three parameters, each corresponding to a respective nonlinear term, that 
depend on a particular focal setting for a lens or lens system. 

Regardless of the particular radial distortion model used, the distortion SR (as given by 
E, (8), for example) may be resolved into *- and, -components so ma, radial distortion 
effects may be accounted for by modifying Eos. (1) and (2). In particular, using the radtal 
distortion model of Eo, (8), accounting for the effects of radial distortion in a cameramodel 
would introduce three parameters (e.g., *„ ft and K,), in addition to the interior orientatton 
pammeters, that may be used to modify Eos. (1) and (2) so as to more accurately represent a 
camera mode!. Some manufacturers of metric cameras may provide such radial distorts 
parameters for different focal settings. Alternatively, such parameters may be determmed 
through camera calibration, as discussed below. 

Another type of distortion effect is •tangential" (or "decentering") lens distortion. 
Tangential distortion refers to a displacement of an image point in ,h. image plane caused by 
misalignment of focussing dements of the lens system, .n conventional photogrammetty 
technicues, tangentia. distortion sometimes is no, modeled because its contribution typically 
is much smaller than radial dis,ortion. Hence, accounting for the effec,s of tangential 
distortion typically is necessaty only for the highest accuracy measurements; in such cases, 
parameters related to tangential distortion also may be used to modify Eos. (1) and (2, so as to 
more accurately represent a camera model. 

In sum, a number of interior orientation and lens distortion parameters may be 
included in a camera model to more accurately represent the projection of an object point of 
interest in a scene onto an image plane of an image recording device. For example, m a 
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digita. device, four interior orientation parameters 0*. distance, - - » 

Jordinates of *e principal pen,, and aspect ratio, and three radia, ,ens portion parameters 
0 e K, *> and K 3 B* TO ) -y be included in a camera model, depending o„ the 

may inciude various interior orientation and ,ens distortion parameters, the „o«a,,on of E,. (5) 
is used to express modified versions of Eos. (1) and (2) in terms of a coordtnate 
transformation function, given by 

•P. -'J {'Pa). <9> 
where •? represents the two (,- and v-> coordinates of the image point a in the image piane, 

coordinate system shown in Fig. 1, and me transformation function ? represents a mapping 
0. acameramodeOftomthethree-dimensionalcameracoordinatesystemtothetwo- 

d— a! image plane. The transformation .unction 7 — - consideration at ,eas, me 
principal distance of the camera, and option* may include terms related to other interior 
orientation and lens distortion parameters, as discussed above, depending on the desired 

accuracy of the camera model. 

p Determining Camera Modeling Parameters via Resection 

From Eq, (6) and (9), the collinearity equations used in resection (discussed above m 

• „f the Related Art) to relate the coordinates of the object point A 
Section C of the Description of the Related An; xo 

in the reference coordinate system of Fig. 1 to image coordinates of the image point a » the 
image p,ane 24 may be rewritten as a coordinate transformation, given by the expre S s,„n 

',T{' r T{'P A ))- (10) 



,t should be appreciated that the transformation given by Ec, (10) represents two collineanty 
canons for the image point „ in the image plane (i.e., one equation for the ,-coordinate and 
oneeouauonformev-coordinate). The transforation taction ? includes the „ 
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parameters of the camera exterior orientation, and the transformation function 7 (i.e., the 
camera model) may include a number of parameters related to the camera interior orienm.cn 
and lens distortion (e.g., four interior orientation parameters, three radial distortion 
parameters, and possibly tangential distortion parameters). As discussed above, the number of 
parameters included in the camera model 'J may depend on the desired level of measurement 
accuracy in a particular photogrammetry application. 

Some or all of the interior orientation and lens distortion parameters of a given camera 
may be known a priori (e.g., from a metric camera manufacturer) or may be unknown (e.g., 
for non-metric cameras). If these parameters are known with a high degree of accuracy (i.e., 
j is reliably known), less rigorous conventional resection methods may be employed based 
on Eq. (10) (e.g., direct evaluation of a system of collinearity equations corresponding to as 
few as tee control points) to obtain the six camera exterior orientation parameters with 
reasonable accuracy. Again, as discussed above in Section D of the Description of the 
Related Art, using a greater number of well-distributed control points and an accurate camera 
model typically improves the accuracy of the exterior orientation parameters obtained by 
conventional resection methods, in that there are more equations in the system of equations 

than there are unknowns. 

If, on the other hand, some or all of the interior orientation and lens distortion 
parameters are not known, they may be reasonably estimated a priori or merely not used in 
the camera model (with the exception of the principal distance; in particular, it should be 
appreciated that, based on the central perspective projection model of Fig.l, at least the 
principal distance must be known or estimated in the camera model J ). Usingacamera 
model > C T that includes fewer and/or estimated parameters generally decreases the accuracy of 
the exterior orientation parameters obtained by resection. However, the resulting accuracy 
may nonetheless be sufficient for some photogrammetry applications; additionally, such 
estimates of exterior orientation parameters may be useful as initial values in an iterative 
estimation process, as discussed above in Section D of the Description of the Related Art. 
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Alternatively, if a more accurate camera model j is desired that includes several 
interior orientation and lens distortion parameters, but some of these parameters are unknown 
or merely estimated a priori, a greater number of control points may be used in some 
conventional resection methods to determine both the exterior orientation parameters as well 
as some or all of the camera model parameters from a single image. Using conventional 
resection methods to determine camera model parameters is one example of "camera 
calibration." 

In camera calibration by resection, the number of parameters to be evaluated by the 
resection method typically determines the number of control points required for a closed-form 
solution to a system of equations based on Eq. (10). It is particularly noteworthy ft* for a 
closed-form solution to a system of equations based on Eq. (10) in which all of the camera 
model and exterior orientation parameters are unknown (e.g., up to 13 or more unknown 
parameters), the control points cannot be co-planar (i.e., the control points may not all lie in a 
same plane in the scene) (see, for example, chapter 3 of the text Three-dimensional Computer 
Vision: A Geometric Viewpoint, written by Olivier Faugeras, published in 1993 by the MIT 
Press, Cambridge, Massachusetts, ISBN 0-262-06158-9, hereby incorporated herein by 
reference). 

In one example of camera calibration by resection, the camera model ' e T may include 
at least one estimated parameter for which greater accuracy is desired (i.e., the principal 
distance of the camera). Additionally, with reference to Eq. (10), there are six unknown 
parameters of exterior orientation in the transformation < r T , thereby constituting a total of 
seven unknown parameters to be determined by resection in this example. Accordingly, at 
least four control points (generating four expressions similar to Eq. (10) and, hence, eight 
collinearity equations) are required to evaluate a system of eight equations in seven 
unknowns. Similarly, if a complete interior orientation calibration of a digital recording 
device is desired (i.e., there are four unknown or estimated interior orientation parameters a 
priori), a total often parameters (four interior and six exterior orientation parameters) need to 
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* determined W -Cion. Accordingly, at leas, five cn.ro> points (general five 

• \ to Ea (10) and hence, .en coffinearity equations) are required .o evaluate 
expressions similar to Eq.(lU) ana, nen 

asv^of.enequauonsin.enunkno^u.ingconvenfionalresecuonrnenrods. 

■ • «,;,™nfthedillital device is unknown, a total of thirteen 
for example, and the extenor orientation ot the digital u 

tor example, orientation parameters, 

parameters need to be determined by resection; namely, six exter 
our interior orientation parameters, and three radia, distortion palters from Eq. (8). 

fourteen equations in thirteen unknowns using conventional resection methods. 



G Intersection 

to exoress the three-dimensional coordinates of the object 
Ea CIO) may be rewritten to express uic u 

a as 



system to the reference coordinate system. Eq. (1 1) represents one of the prim^ goals of 
eonventionalphotogrammetry techniques; ^^^^^ 

0! *° ^discussed above in Section B of the Description of the Related MU however, a 
.osed-formsoluUontoEq.Cl^maynotbedeterminedmere.yfromd.emeasuredimage 
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coordinates <P„of a single image poin, «. even if the exterior orientation parameters m 7 and 
the camera mode! ;r are known with any degree of accuracy. This is because Eq. (1 1) 
essentially represents two eollinearity equations based on the fundamental relationships g,ven 
in Eqs (1) and (2), but there are three unknowns in the two equations (i.e., the three 
coordinates of the object poinU). In particular, the taction j- <</>„) in Eq. (1 1) has no 
closed-form solution unless more information is known (e.g., "depth" information, such as a 
distance from the camera origin to (he object poin, A). For this reason, conventional 
photogrammetry techniques require a, !eas« two different images of a scene in which an object 
point of interest is present to determine the three-dimensional coordinates in the scene of the 
object point. This process commonly is referred to in photogrammetry as "intersection ." 

With reference to Fig. 3, if the exterior orientation and camera model parameters of 
two cameras represented by the coordinate systems 76, and 76 2 are known (e.g., previously 
determined from two independent resections with respect to a common reference coordmate 
system 74), the three-dimensional coordinates 'P A of the object poin. A in the reference 
coordinate system 74 can be evaluated from the image coordinates "P„, of a firs, image pomt 
a, (51-,) in the image plane 24, of a first camera, and from the image coordinates"^ of a 
second image point a, <SVt> in the image plane 24, of a second camera. In this case, an 
expression similar to Eq. (1 1) is generated for each image point a, and a> each express.on 
representing Wo eollinearity equations; hence, the Wo different images of tire object point A 
give rise to a system of four eollinearity equations in three unknowns. 

As with resection, the intersection method used to evaluate such a system of equations 
depends on the degree of accuracy desired in the coordinates of tire object point A. For 
example, conventional intersection methods are known for direct evaluation of the system of 
collineartty equations from two different images of the same point. For higher accuracy, a 
linearized iterative leas, squares estimation process may be used, as discussed above. 

Regardless of the particular intersection method employed, independent resections of 
wo cameras followed by intersections of object points of interest in a scene using 



course 
common 
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corresponding images of the object points are common procedures in photogrammetry. Of 
it should be appreciated that the independent resections should be with respect to a 
reference coordinate system for the scene. In a case where a number of control 
points (i.e., at least three) are chosen in a scene for a given resection (e.g., wherein at least 
some of the control points may define the reference coordinate system for the scene), 
generally the control points need to be carefully selected such that they are visible in images 
taken by cameras at different locations, so that the exterior orientation of each camera may be 
determined with respect to a common reference coordinate system. As discussed above in 
Section D of the Description of the Related Art, choosing such control points often is not a 
trivial task, and the reliability and accuracy of multi-camera resection followed by intersection 
may be vulnerable to analyst errors in matching corresponding images of the control points in 
the multiple images. 

H. Multi-image Photogrammetry and "Bundle Adjustments " 
Fig. 4 shows a number of cameras at different locations around an object of interest, 
represented by the object point A. While Fig. 4 shows five cameras for purposes of 
illustration, any number of cameras may be used, as indicated by the subscripts 1, 2, 3...J. For 
example, the coordinate system of theyth camera is indicated in Fig. 4 with the reference 
character 76; and has an origin O c , Similarly, an image point corresponding to the object 
points obtained by the/th camera is indicated as a/m the respective image plane 24,. Each 
image point a, - aj is associated with two collinearity equations, which may be alternatively 
expressed (based on Eqs. (10) and (1 1), respectively) as 



= jrtfrra) (12) 



or 



(13) 
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As discussed above, the coUinearity equations represented by E,s. (12, and (13) each include 
six parameters for ,he exterior orientation of a particular camera (in ? ), as well as various 
camera mode, parameters (e.g., interior orientation, lens distortion) for the particular camera 
On n -) Accordingly, for a total a J cameras, i, should be appreciated mat a number; of 
expressions each given by Eqs. (12) or (13) represent a system of 2, collinearity stations for 
the object point A, wherein the system of collinearity equations may have various known and 
unknown parameters. 

A generalized functional model for muM-image photogrammetry based on a system of 
equations derived from either of Eqs. (.2) or (13, for anumber of object points of interest .n a 
scene may be given by the expression 

F(U,V,W) = °. (14> 
where V is a vector representing unknown parameters in the system of equations (i.e., 
parameters whose values are desired), V is a vector representing measured parameters, and W 
is a vector representing known parameters. Stated differently, the expression of Eq. (14) 
represents art evaluation of a system of collinearity equations for parameter values m Ore 
vector U, given parameter values for the vectors Kand W. 

Generally, in multi-image photogrammetry, choices must be made as to which 
peters are known or estimated (for the vector W>. which parameters are measured (for the 
vector n ^ which parameters are to be determined (in the vector U). For example, rn 
some applications, tire vector Fmay include all measured image coordinates of ,h. 
corresponding image points for each object point of interest, and also may include the 
coordinates in the reference coordinate system of any control points in the scene, if known. 
Likewise, the three-dimensional coordinates of object points of interest in the reference 
coordinate system may be included in tire vector U as unknowns. If the cameras have each 
undergone prior calibration, and/or accurate, reliaWe values are known for some or all of the 
camera model parameters, these parameters may be inchrded in tire vector W as known 
constant, Alternatively, if no prior values for the camera model parameters have been 
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obtained, it is possible to include these parameters in the vector U as unknowns. For example, 
exterior orientation parameters of the cameras may have been evaluated by a prior resection 
and can be included as either known constants in the vector ^or as measured or reasonably 
estimated parameters in the vector V, so as to provide for the evaluation of camera model 
parameters. 

The process of simultaneously evaluating, from multiple images of a scene, the three- 
dimensional coordinates of a number of object points of interest in the scene and the exterior 
orientation parameters of several cameras using least squares estimation based on a system of 
collinearity equations represented by the model of Eq. (14) commonly is referred to in 
photogrammetry as a "bundle adjustment." When parameters of the camera model (e.g., 
interior orientation and lens distortion) are also evaluated in this manner, the process often is 
referred to as a "self-calibrating bundle adjustment." For a multi-image bundle adjustment, 
generally at least two control points need to be known in the scene (more specifically, a 
distance between two points in the scene) so that a relative scale of the reference coordinate 
system is established. In some cases, based on the number of unknown and known (or 
measured) parameters, a closed-form solution for U in Eq. (14) may not exist. However, an 
iterative least squares estimation process may be employed in a bundle adjustment to obtain a 
solution based on initial estimates of the unknown parameters, using some initial constraints 
for the system of collinearity equations. 

For example, in a multi-image bundle adjustment, if seven unknown parameters 
initially are assumed for each camera that obtains a respective image (i.e., six exterior 
orientation parameters and the principal distance d for each camera), and three unknown 
parameters are assumed for the three-dimensional coordinates of each object point of interest 
in the scene that appears in each image, a total of 7/ + 3 unknown parameters initially are 
assumed for each object point that appears in / different images. Likewise, as discussed 
above, each object point in the scene corresponds to 2j collinearity equations in the system of 
equations represented by Eq. (14). To arrive at an initial closed-form solution to Eq. (14), the 
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number of equations in the system shouVd be greater or equal to the number of unknown 
parameter, According*, for the foregoing examp.e, a constrain, —hip for the system 
of equations represented by Eq. (14) may be given by 

2;»2 7;+3n, <15) 

where n is the number of object points of interest in the scene that each appears in; different 
image, For example, using the constrain, relationship given by Eq. (15), an initial closed- 
form solution to Eq. (14) may be obtained using seven control points („ - 7) and mree 
different images (/ - 3), to give a system of 42 collinearity equations in 42 unknowns. It 
should be appreciated that if more (or less) than seven unknown parameters are hutrally 
assumed for each camera, the constant multiplying the variable; on the right side of Eq. (.5) 
changes accordingly, .n particular, a generalized constrain, relationship that apphes ,o bo* 
bundle and self-calibrating bundle adjustments may be given by 

2jn 2 C; + 3rt , 

wh ere C indicates tire total number of initially assumed unknown exterior orientation and/or 
camera model parameters for each camera. 

Generally, a multi-image bundle (or self calibrating bundle) adjustment accordmg to 
Eq (14) gives resuhs of higher accuracy than resection and intersection, bu, a. a cos,. For 
example, tire constrain, relationship of Eq. (16) implies tha, some minimum number of camera 
,„cations must be used to obtain multiple (i.e., different) images of some minimum number of 
object points of interest in the scene for the determination of unknown parameters usrng a 
bundle adjustmentproces, In particuiar, with reference to Eq. (16), inabund.e adjustment, 
typically an analyst mus, select some number „ of objectpoints of interest in tire scene that 
each appear in some number; of different images of the scene, and correctly match; 
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copending image points of each respective object point from image to image. For 
purposes of the present disdosure, the process of matching corresponding image potnts of an 
object point mat appear in multiple images is referred to as "referencing." 

in a bundle adjustment, once the image points are "referenced" by an analyst m me 
multiple images for each object point, typically all measured image coordinates of the 
referenced image points for all of the object points are processed simultaneously as measured 
parameters inthe vector Kof the model of Eq. (14) to evaluate exterior orientation and 
perhaps camera mode, parameters, as weU as the three^hnensional coordinates of each object 
point (which would be elements of the vector Win mis case). Accordingly, i, may be 
appreciated tha, the simultaneous solution in a bundle adjustment process of the system of 
equations modeled by E q . (14) typically involves large data sets and the computation of 

inverses of large matrices. 

One noteworthy issue with respect to bundle adjustments is mat the ten*, 
estimation process makes it difficult to identify errors in any of the measured parameters used 
in the vector K of the mode, of Eq. (14), due to the large data sets involved in the system of 
severa, equations. For example, if an analyst makes an error during the referencing process 
(e g the analyst fails to correctly match, or "reference," an image point a, of a firs, object 
point A in a first image to an image point a 2 of the first object point A in a second image, and 
instead references the image point a, to an image point of a second object point B » the 
second image), the bundle adjustment process wi,l produce erroneous results, the source of 
which may be quite difficult to trace. An ar^ys, error in referencing (matching) image pomts 
of an object point in muitiple images commonly is referred to in photogrammetry as a 
"blunder " While the constraint relationship of Eq. (1 6) suggests tha, more object potnts and 

bundle adjustment process, me need to reference a greater number of object points as they 
appear in a greater number of images may in some cases increase the probability of analyst 
blunder, and hence decrease the reliability of the bundle adjustment results. 
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/. Summary 

From the foregoing discussion, it should be appreciated that conventional 
photogrammetry techniques generally involve obtaining multiple images (from different 
locations) of an object of interest in a scene, to determine from the images actual three- 
dimensional position and size information about the object in the scene. Additionally, 
conventional photogrammetry techniques typically require either specially manufactured or 
adapted image recording devices (generally referred to herein as "cameras"), for which a 
variety of calibration information is known a priori or obtained via specialized calibration 
techniques to insure accuracy in measurements. 

Furthermore, a proper application of photogrammetry methods often requires a 
specialized analyst having training and knowledge, for example, in photo-surveying 
techniques, optics and geometry, computational processes using large data sets and matrices, 
etc. For example, in resection and intersection processes (as discussed above in Sections D, F, 
and G of the Description of the Related Art), typically an analyst must know actual relative 
position and/or size information in the scene of at least three control points, and further must 
identify (i.e., "reference") corresponding images of the control points in each of at least two 
different images. Alternatively, in a multi-image bundle adjustment process (as discussed 
above in Section H of the Description of the Related Art), an analyst must choose at least two 
control points in the scene to establish a relative scale for objects of interest in the scene. 
Additionally, in a bundle adjustment, an analyst must identify (i.e., "reference") often several 
corresponding image points in a number of images for each of a number of objects of interest 
in the scene. This manual referencing process, as well as the manual selection of control 
points, may be vulnerable to analyst errors or "blunders," which lead to erroneous results in 
either the resection/intersection or the bundle adjustment processes. 

Additionally, conventional photogrammetry applications typically require 
sophisticated computational approaches and often require significant computing resources. 
Accordingly, various conventional photogrammetry techniques generally have found a 
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somewhat limited application by specialized practitioners and analysts (e.g., scientists, 
military personnel, etc.) who have the availability and benefit of complex and often expense 
equipment and instrumentation, significant computational resources, advanced trammg, and 
the like. 

Summary of the Invention 
One embodiment of the invention is directed to an image metrology reference target, 
comprising a, lest one fiducial mark, and a, leas, one orientation dependent radiation source 
disposed ^predetermined spatial relationship with respect ,0 the a, leas, one fiducial mark. 
The at least one orientation dependent radiation source emanates, from an observation surface, 
orientationdependent radiation having at least one detectable property in an image of *e 
reference target mat varies as a function of a, least one of a rotation angle of the orientation 
dependent radiation source and a distance between the orientation dependent radiation source 
and a camera obtaining the image of the reference target. 

Another embodiment of the invention is directed to an apparatus, comprising at least 
one orientation dependent radiation source to emanate, from an observation surface, 
orientation dependent radiation having at least one detectable property that varies as a 
function of a, leas, one of a rotation angle of die orientation dependent radiation source and a 
distance between the orientation dependent radiation source and a radiation detection devrce 
receiving the orientation dependent radiation. 

Another embodiment of the invention is directed to a method for processing an image. 
The image includes at leas, one orientation dependent radiation source that emanates, from an 
observation surface, orientation dependent radiation having at leas, a first detectable property 
in the image and a second detectable property in the image mat each varies as a function of a. 
least one of a rotation angle of the orientation dependent radiation source and a distance 
between the orientation dependent radiation source and a camera obtaining the image of the a. 
leas, one orientation dependent radiation source. The method comprises acts of determinmg 
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the rotation angle of the orientation dependent radiation source from the first detectable 
property, and determining the distance between the orientation dependent radiation source and 
the camera from at least the second detectable property. 

Another embodiment of the invention is directed to a computer readable medium 
encoded with a program for execution on at least one processor. The program, when executed 
on the at least one processor, performs a method for processing an image. The image includes 
at least one orientation dependent radiation source that emanates, from an observation surface, 
orientation dependent radiation having at least a first detectable property in the image and a 
second detectable property in the image that each varies as a function of at least one of a 
rotation angle of the orientation dependent radiation source and a distance between the 
orientation dependent radiation source and a camera obtaining an image of the at least one 
orientation dependent radiation source. The method executed by the program comprises acts 
of determining the rotation angle of the orientation dependent radiation source from the first 
detectable property, and determining the distance between the orientation dependent radiation 
source and the camera from at least the second detectable property. 

Another embodiment of the invention is directed to a method in a system including at 
least one orientation dependent radiation source that emanates, from an observation surface, 
orientation dependent radiation having at least a first detectable property and a second 
detectable property that each varies as a function of at least one of a rotation angle of the 
orientation dependent radiation source and a distance between the orientation dependent 
radiation source and a radiation detection device receiving the orientation dependent radiation. 
The method comprises acts of determining the rotation angle of the orientation dependent 
radiation source from the first detectable property, and determining the distance between the 
orientation dependent radiation source and the radiation detection device from at least the 

second detectable property. 

Another embodiment of the invention is directed to a computer readable medium 
encoded with a program for execution on at least one processor. The program, when executed 
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on 



the at least one processor, performs a method in a system including at least one orientation 
dependent radiation source that emanates, from an observation surface, orientation dependent 
radiation having at least a first detectable property and a second detectable property that each 
varies as a function of at least one of a rotation angle of the orientation dependent radiation 
source and a distance between the orientation dependent radiation source and a radiation 
detection device receiving the orientation dependent radiation. The method executed by the 
program comprises acts of determining the rotation angle of the orientation dependent 
radiation source from the first detectable property, and determining the distance between the 
orientation dependent radiation source and the radiation detection device from at least the 

second detectable property. 

Another embodiment of the invention is directed to an image metrology reference 
target, comprising automatic detection means for facilitating an automatic detection of the 
reference target in an image of the reference target obtained by a camera, and bearing 
determination means for facilitating a determination of at least one of a position and at least 
one orientation angle of the reference target with respect to the camera. 

Brief Description of the Drawings 

The accompanying drawings are not intended to be drawn to scale. In the drawings, 
each identical or nearly identical component that is illustrated in various figures is represented 
by a like reference character. For purposes of clarity, not every component may be labeled in 
every drawing. 

Fig. 1 is a diagram illustrating a conventional central perspective projection imaging 
model using a pinhole camera; 

Fig. 2 is a diagram illustrating a coordinate system transformation between a reference 
coordinate system for a scene of interest and a camera coordinate system in the model of 
Figl; 

Fig. 3 is a diagram illustrating the concept of intersection as a conventional 
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photogrammetry technique; 

Fig. 4 is a diagram illustrating the concept of conventional multi-image 

photogrammetry; 

Fig. 5 is a diagram illustrating an example of a scene on which image metrology may 
be performed using a single image of the scene, according to one embodiment of the 

invention; 

Fig. 6 is a diagram illustrating an example of an image metrology apparatus according 

to one embodiment of the invention; 

Fig. 7 is a diagram illustrating an example of a network implementation of an image 
metrology apparatus according to one embodiment of the invention; 

Fig. 8 is a diagram illustrating an example of the reference target shown in the 
apparatus of Fig. 6, according to one embodiment of the invention; 

Fig. 9 is a diagram illustrating the camera and the reference target shown in Fig. 6, for 
purposes of illustrating the concept of camera bearing, according to one embodiment of the 
invention; 

Fig. lOAisadiagramillustratingarear view of the reference target shown inFig. 8, 

according to one embodiment of the invention; 

Fig. 1 OB is a diagram illustrating another example of a reference target, according to 

one embodiment of the invention; 

Fig. IOC is a diagram illustrating another example of a reference target, according to 

one embodiment of the invention; 

Figs. 1 1 A-l 1C are diagrams showing various views of an orientation dependent 
radiation source used, for example, in the reference target of Fig. 8, according to one 

embodiment of the invention; 

Figs. 12A and 12B are diagrams showing particular views of the orientation dependent 
radiation source shown in Figs. 1 1 A- 11C, for purposes of explaining some fundamental 
concepts according to one embodiment of the invention; 
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Figs. 13A-13D are graphs showing plots of various radiation transmission 
characteristics of the orientation dependent radiation source of Figs. 1 1 A-l 1C, according to 
one embodiment of the invention; 

Fig. 14 is a diagram of landmark for machine vision, suitable for use as one or more of 
the fiducial marks shown in the reference target of Fig. 8, according to one embodiment of the 
invention; 

Fig. 15 is a diagram of a landmark for machine vision according to another 

embodiment of the invention; 

Fig. 16A is a diagram of a landmark for machine vision according to another 

embodiment of the invention; 

Fig. 16B is a graph of a luminance curve generated by scanning the mark of Fig. 16A 
along a circular path, according to one embodiment of the invention; 

Fig. 16C is a graph of a cumulative phase rotation of the luminance curve shown in 
Fig 16B, according to one embodiment of the invention; 

Figs. 17A is a diagram of the landmark shown in Fig. 16A rotated obliquely with 
respect to the circular scanning path; 

Fig. 17B is a graph of a luminance curve generated by scanning the mark of Fig. 17A 
along the circular path, according to one embodiment of the invention; 

Fig.. 17C is a graph of a cumulative phase rotation of the luminance curve shown in 
Fig. 17B, according to one embodiment of the invention; 

Fig. 18A is a diagram of the landmark shown in Fig 16A offset with respect to the 

circular scanning path; 

Fig. 18B is a graph of a luminance curve generated by scanning the mark of Fig. 87 A 
along the circular path, according to one embodiment of the invention; 

Fig. 18C is a graph of a cumulative phase rotation of the luminance curve shown in 
Fig. 18B, according to one embodiment of the invention; 

Fig. 19 is a diagram showing an image that contains six marks similar to the mark 
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shown in Fig. 16A, according to one embodiment of the invention; 

Fig. 20 is a graph showing a plot of individual pixels that are sampled along the 
circular path shown in Figs. 16A, 17A, and 18A, according to one embodiment of the 

invention; 

Fig. 21 is a graph showing a plot of a sampling angle along the circular path of Fig. 20, 
according to one embodiment of the invention; . 

Fig 22A is a graph showing a plot of an unfiltered scanned signal representing a 
random luminance curve generated by scanning an arbitrary portion of an image that does not 
contain a landmark, according to one embodiment of the invention; 

Fig. 22B is a graph showing a plot of a filtered version of the random luminance curve 

shown in Fig. 22A; 

Fig. 22C is a graph showing a plot of a cumulative phase rotation of the filtered 
luminance curve shown in fig. 22B, according to one embodiment of the invention; 

Fig. 23A is a diagram of another robustmark according to one embodiment of the 

invention; 

Fig. 23B is a diagram of the mark shown in Fig. 23A after color filtering, according to 

one embodiment of the invention; 

Fig. 24A is a diagram of another fiducial mark suitable for use in the reference target 
shown in Fig. 8, according to one embodiment of the invention; 

Fig. 24B is a diagram showing a landmark printed on a self-adhesive substrate, 
according to one embodiment of the invention; 

Figs. 25 A and 25B are diagrams showing a flow chart of an image metrology method 
according to one embodiment of the invention; 

Fig. 26 is a diagram illustrating multiple images of differently-sized portions of a 
scene for purposes of scale-up measurements, according to one embodiment of the invention; 

Figs. 27-30 are graphs showing plots of Fourier transforms of front and back gratings 
of an orientation dependent radiation source, according to one embodiment of the invention; 
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Figs. 3 1 and 32 are graphs showing plots of Fourier transforms of radiation emanated 
from an orientation dependent radiation source, according to one embodiment of the 

invention; 

Fig. 33 is a graph showing a plot of a triangular waveform representing radiation 
emanated from an orientation dependent radiation source, according to one embodiment of the 

invention; 

Fig. 34 is a diagram of an orientation dependent radiation source according to one 
embodiment of the invention, to facilitate a far-field observation analysis; 

Fig. 35 is a graph showing a plot of various terms of an equation relating to the 
determination of rotation or viewing angle of an orientation dependent radiation source, 
according to one embodiment of the invention; 

Fig. 36 is a diagram of an orientation dependent radiation source according to one 
embodiment of the invention, to facilitate a near-field observation analysis; 

Fig. 37 is a diagram of an orientation dependent radiation source according to one 
embodiment of the invention, to facilitate an analysis of apparent back grating shift in the 
near-field with rotation of the source; 

Fig. 38 is a diagram showing an image including a landmark according to one 
embodiment of the invention, wherein the background content of the image includes a number 
of rocks; 

Fig. 39 is a diagram showing a binary black and white threshold image of the image 
of Fig. 38; 

Fig. 40 is a diagram showing a scan of a colored mark, according to one embodiment 
of the invention; 

Fig. 41 is a diagram showing a normalized image coordinate frame according to one 

embodiment of the invention; and 

Fig. 42 is a diagram showing an example of an image of fiducial marks of a reference 
target to facilitate the concept of fitting image data to target artwork, according to one 
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embodiment of the invention. 

Detailed Description 

A. Overview 

As' discussed above in connection with conventional photogrammetry techniques, 
determining position and/or size information for objects of interest in a three-dimensional 
scene from two-dimensional images of the scene can be a complicated problem to solve. In 
particular, conventional photogrammetry techniques often require a specialized analyst to 
know some relative spatial information in the scene a priori, and/or to manually take some 
measurements in the scene, so as to establish some frame of reference and relative scale for 
the scene. Additionally, in conventional photogrammetry techniques, multiple images of the 
scene (wherein each image includes one or more objects of interest) generally must be 
obtained from different respective locations, and often an analyst must manually identify 
corresponding images of the objects of interest that appear in the multiple images. This 
manual identification process (referred to herein as "referencing") may be vulnerable to 
analyst errors or "blunders," which in turn may lead to erroneous results for the desired 
information. 

Furthermore, conventional photogrammetry techniques typically require sophisticated 
computational approaches and often require significant computing resources. Accordingly, 
various conventional photogrammetry techniques generally have found a somewhat limited 
application by specialized practitioners who have the availability and benefit of complex and 
often expensive equipment and instrumentation, significant computational resources, 

advanced training, and the like. 

In view of the foregoing, various embodiments of the present invention generally 
relate to automated, easy-to-use, image metrology methods and apparatus that are suitable for 
specialist as well as non-specialist users (e.g., those without specialized training in 
photogrammetry techniques). For purposes of this disclosure, the term "image metrology" 
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generally refers to the concept of image analysis for various measurement purposes. 
Similarly, for purposes of illustration, some examples of "non-specialist users" include, but 
are not limited to, general consumers or various non-technical professionals, such as 
architects, building contractors, building appraisers, realtors, insurance estimators, interior 
designers, archaeologists, law enforcement agents, and the like. In one aspect of the present 
invention, various embodiments of image metrology methods and apparatus disclosed herem 
in general are appreciably more user-friendly than conventional photogrammetry methods and 
apparatus. Additionally, according to another aspect, various embodiments of methods and 
apparatus of the invention are relatively inexpensive to implement and, hence, generally more 
affordable and accessible to non-specialist users than are conventional photogrammetry 

systems and instrumentation. 

Although one aspect of the present invention is directed to ease-of-use for non- 
specialist users, it should be appreciated nonetheless that image metrology methods and 
apparatus according to various embodiments of the invention may be employed by specialized 
users (e.g., photogrammetrists) as well. Accordingly, several embodiments of the present 
invention as discussed further below are useful in a wide range of applications to not only 
non-specialist users, but also to specialized practitioners of various photogrammetry 
techniques and/or other highly-trained technical personnel (e.g., forensic scientists). 

In various embodiments of the present invention related to automated image metrology 
methods and apparatus, particular machine vision methods and apparatus according to the 
invention are employed to facilitate automation (i.e., to automatically detect particular 
features of interest in the image of the scene). For purpose of this disclosure, the term 
"automatic" is used to refer to an action that requires only minimum or no user involvement. 
For example, as discussed further below, typically some minimum user involvement is 
required to obtain an image of a scene and download the image to a processor for processing. 
Additionally, before obtaining the image, in some embodiments the user may place one or 
more reference objects (discussed further below) in the scene. These fundamental actions of 
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acquiring and downloading an image and placing one or more reference objects in the scene 
are considered for purposes of this disclosure as minimum user involvement. In view of the 
foregoing, the term "automatic" is used herein primarily in connection with any one or more 
of a variety of actions that are carried out, for example, by apparatus and methods according 
to the invention which do not require user involvement beyond the fundamental actions 
described above. 

In general, machine vision techniques include a process of automatic object 
recognition or "detection," which typically involves a search process to find a correspondence 
between particular features in the image and a model for such features that is stored, for 
example, on a storage medium (e.g., in computer memory). While a number of conventional 
machine vision techniques are known, Applicants have appreciated various shortcomings of 
such conventional techniques, particularly with respect to image metrology applications. For 
example, conventional machine vision object recognition algorithms generally are quite 
complicated and computationally intensive, even for a small number of features to identify in 
an image. Additionally, such conventional algorithms generally suffer (i.e., they often 
provide false-positive or false-negative results) when the scale and orientation of the features 
being searched for in the image are not known in advance (i.e., an incomplete and/or 
inaccurate correspondence model is used to search for features in the image). Moreover, 
variable lighting conditions as well as certain types of image content may make feature 
detection using conventional machine vision techniques difficult. As a result, highly 
automated image metrology systems employing conventional machine vision techniques 
historically have been problematic to practically implement. 

However, Applicants have identified solutions for overcoming some of the difficulties 
typically encountered in conventional machine vision techniques, particularly for application 
to image metrology. Specifically, one embodiment of the present invention is directed to 
image feature detection methods and apparatus that are notably robust in terms of feature 
detection, notwithstanding significant variations in scale and orientation of the feature 




-39- 



searched for in the image, lighting conditions, camera settings, and overall image content, for 
example. In one aspect of this embodiment, feature detection methods and apparatus of the 
invention additionally provide for less computationally intensive detection algorithms than do 
conventional machine vision techniques, thereby requiring less computational resources and 
providing for faster execution times. Accordingly, one aspect of some embodiments of the 
present invention combines novel machine vision techniques with novel photogrammetry 
techniques to provide for highly automated, easy-to-use, image metrology methods and 
apparatus that offer a wide range of applicability and that are accessible to a variety of users. 

In addition to automation and ease-of-use, yet another aspect of some embodiments of 
the present invention relates to image metrology methods and apparatus that are capable of 
providing position and/or size information associated with objects of interest in a scene from a 
single image of the scene. This is in contrast to conventional photogrammetry techniques, as 
discussed above, which typically require multiple different images of a scene to provide three- 
dimensional information associated with objects in the scene. It should be appreciated that 
various concepts of the present invention related to image metrology using a single image and 
automated image metrology, as discussed above, may be employed independently in different 
embodiments of the invention (e.g., image metrology using a single image, without various 
automation features). Likewise, it should be appreciated that at least some embodiments of 
the present invention may combine aspects of image metrology using a single image and 
automated image metrology. 

For example, one embodiment of the present invention is directed to image metrology 
methods and apparatus that are capable of automatically determining position and/or size 
information associated with one or more objects of interest in a scene from a single image of 
the scene. In particular, in one embodiment of the invention, a user obtains a single digital 
image of the scene (e.g., using a digital camera or a digital scanner to scan a photograph), 
which is downloaded to an image metrology processor according to one embodiment of the 
invention. The downloaded digital image is then displayed on a display (e.g., a CRT monitor) 
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coupled .o the processor. .„ one aspect of this embodiment, the user indicates one or more 
points of interest in rhe scene via the displayed ima 6 e using a user interface coupled to the 

identifies points of interest that appear in the digital image of the scene using feature de.ect.on 

the image to automatic* determine various camera calibration information, and ulttmately 
determines position and/or size information associated with the indicated or automata 
identified point or points of interest in the scene, m sum, the user obtains a single unage of 
the scene, downloads the image ,„ the processor, and easily obtains position and/or s IZ e 
information associated with objects of interest in the scene. 

,„ some embodiments of the present invention, the scene of interest includes one or 
more reference objects ma, appear in an image of the scene. For purposes of mis disclosure, 
the term "reference object" genera!* refers to an object in the scene for which at leas, one or 
more of size (dimensional), spa,ial position, and orientation informal is knowna^n 
with respec, to a reference coordina,e system for ,he scene. Various information Known a 
prl0 r, in connect with one or more reference objects in a scene is referred ,o herem 
generally as "reference information." 

According ,o one embodiment, one example of a reference object is given by a control 
point which, as discussed above, is a point in the scene whose three-dimensiona, coordinates 
a^ownwimrespecUoareference coordinate system forme scene. In this example, the 
^-dimensional coordinates of me control poin, constitute the reference informal 

object" as used herein is not limited merely to the foregoing example of a control pom bu, 

to Known coordinates of control poin., bu. may include other types of informahon, as 
discussed former below. Additionally, according .o some embodiments, i. should be 
appeared that various types of reference objects may themselves establish the reference 
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coordinate system for the scene. 

In general, according to one aspect of the invention, one or more reference objects as 
discussed above in part facilitate a camera calibration process to determine a variety of 
camera calibration information. For purposes of this disclosure, the term "camera calibrate 
information" generally refers to one or more exterior orientation, interior orientation, and lens 
distortion parameters for a given earner. In particular, as discussed above, the camera 
exterior orientation refers to the position and orientation of the camera relative to the scene of 
interest, while the interior orientation and lens distortion parameters in general constitute a 
camera model that describes how a particular camera differs from an idealized pinhole 
earner. According to one embodiment, various camera calibration information is determmed 
based at least in par, on the reference information know aprior, ma, is associated with one 
or more reference objects included in the scene, together with information mat is derived from 
the image of such reference objects in an image of the scene. 

According to one embodiment of me invention, certain types of reference objects are 
included in the scene ,0 facilitate an automated camera calibration process. In particular, in 
one embodiment, one or more reference objects included in a scene of interest may be m the 
form of a "robust fiducial mark" (hereinafter abbreviated as RF ID) that is placed in the scene 
before an image of tire scene is taken, such that the RFID appears in the image. For purposes 
of the this disclosure, the term "robust fiducial mark" generally refers ,o an object whose 
image has one or more properties that do no, change as a ft.nc.ion of point-of-view, various 
camera settings, different lighting conditions, etc. 

in particular, according to one aspect of this embodiment, the image of an RFID has an 
invariance with respect to scale or tilt; stated differentiy, a robust fiducial mark has one or 
more unique detectable properties in an image that do no, change as a function of either the 
size of me mark as i, appears in the image, or the orient of the mark with respect ,o me 
camera as the image ofthe scene is obtained. In other aspects, an RFID preferably has one or 
more invariant characteristics mat are relatively simple to detect in an image, tha, are unhkely 
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t0 occur by chance in a given scene, and tha, are relatively unaffected by different types of 

general image content. 

In general, the above-described characteristics of one or more RFIDs that are inciuded 
in a scene of interest significantly facilitate automatic feature detection according to various 
embodiments of the invention. In particular, one or more RFIDs tha, are placed in the scene 
as reference objects facilitate an automatic determination of various camera calibrate 
information. However, it should be appreciated that tire use of RFIDs in various embodiments 
of the present invention is not limited to reference objects. 

For example, as discussed further below, one or more RFIDs may be arbitrarily placed 
in the scene to facilitate automatic identification of objects of interest in tire scene for which 
position and/or size information is no, known but desired. Additionally, RFIDs may be 
placed in the scene a. particular locations ,0 establish automatical detectable link points 
between multiple images of a large and/or complex space, for purposes of site surveying usmg 
image metrology methods and apparatus according to the invention. It should be appreciated 
ma, the foregoing examples are provided merely for purposes of illustration, and tha. RFIDs 
have a wide variety of uses in image metrology methods and apparatus according to tire 
invention, as discussed former below. In one embodiment, RFIDs are printed on self- 
adhesive substrates (e.g., self-stick removable notes) which may be easi.y affixed a, desrred 
iocations in a scene prior ,o obtaining one or more images of the scene to facilitate automatic 
feature detection. 

With respect to reference objects, according to another embodiment of tire invention, 
one or more reference objects in the scene may be in the form of an "orientation-dependent 
radiation source" Rafter abbreviated as ODR) ,ha, is placed in tire scene before . rmage 
of me scene is ton, such ,ha, tire ODR appears in tire image. For purposes of this disclosure, 
an orientation-dependent radiation source generally refers to an object mat emanates radiation 
having a, leas, one detectable property, based on an orientation of the object, mat is capable of 
being detected from the image of the scene. Some examples of ODRs suitable for purposes of 
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the present invention include, but are not limited to, devices described in U.S. Patent No. 
5,936,723, dated August 10, 1999, entitled "Orientation Dependent Reflector," hereby 
incorporated herein by reference, and in U.S. Patent Application Serial No. 09/317,052, filed 
May 24, 1999, entitled "Orientation-Dependent Radiation Source," also hereby incorporated 
herein by reference, or devices similar to those described in these references. 

In particular, according to one embodiment of the present invention, the detectable 
property of the radiation emanated from a given ODR varies as a function of at least the 
orientation of the ODR with respect to a particular camera that obtains a respective image of 
the scene in which the ODR appears. According to one aspect of this embodiment, one or 
more ODRs placed in the scene directly provide information in an image of the scene that is 
related to an orientation of the camera relative to the scene, so as to facilitate a determination 
of at least the camera exterior orientation parameters. According to another aspect, an ODR 
placed in the scene provides information in an image that is related to a distance between the 
camera and the ODR. 

According to another embodiment of the invention, one or more reference objects may 
be provided in the scene in the form of a reference target that is placed in the scene before an 
image of the scene is obtained, such that the reference target appears in the image. According 
to one aspect of this embodiment, a reference target typically is essentially planar in 
configuration, and one or more reference targets may be placed in a scene to establish one or 
more respective reference planes in the scene. According to another aspect, a particular 
reference target may be designated as establishing a reference coordinate system for the scene 
(e.g., the reference target may define an x-y plane of the reference coordinate system, wherein 
a z-axis of the reference coordinate system is perpendicular to the reference target). 

Additionally, according to various aspects of this embodiment, a given reference target 
may include a variety of different types and numbers of reference objects (e.g., one or more 
RFIDs and/or one or more ODRs, as discussed above) that are arranged as a group in a 
particular manner. For example, according to one aspect of this embodiment, one or more 
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RFIDs and/or ODRs included in a given reference target have known particular spatial 
relationships to one another and to the reference coordinate system for the scene. 
Additionally, other types of position and/or orientation information associated with one or 
more reference objects included in a given reference targe, may be known a prion; 
accordingly, unique reference information may be associated with a given reference target. 

,„ another aspect of mis embodiment, combinations of RFIDs and ODRs employed » 
reference targets according to the invention facilitate an automatic determination of various 
camera calibration information, including one or more of exterior orientation, interior 
orientation, and lens distortion parameters, as discussed above. Furthermore, in yet another 
aspect, particular combinations and arrangements of RFIDs and ODRs in a reference targe, 
according ,0 tire invention provide for a determination of extensive camera calibrate 
information (including several or all of ,he exterior orientation, interior orientation, and lens 
distortion parameters) using a single planar reference target in a single image. 

While tire foregoing concepts related ,0 image metrology methods and apparatus 
according to the invention have been introduced in part with respect to image metrology ustng 
single-images, it shou.d be appreciated nonetheless that various embodiments of me present 
invention incorporating tire foregoing and other concepts are directed to image metrology 
memoes and apparatus using two or more images, as discussed further below. In particular, 
according to various multi-image embodiments, methods and apparatus of tire present 
invention are capable of automatically tying together multiple images of a scene of .uteres, 
(which in some cases may be too large to capture completely in a single image), to provde for 
three-dimensional image metrology surveying of large and/or complex spaces. Additionally, 
some multi-image embodiments provide for mree-dimensiona! image metrology from stereo 
images, as well as redundant measurements to improve accuracy. 

In yet another embodiment, image mettoiogy methods and apparatus according to the 
present invention may be implemented over a local-area network or a wide-area network, such 
as the interne,, so as ,0 provide image metrology services to a number of network client, In 
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one aspect of this embodiment, a number of system users at respective client workstations 
may upload one or more images of scenes to one or more centralized image metrology servers 
via the network. Subsequently, clients may download position and/or size information 
associated with various objects of interest in a particular scene, as calculated by the server 
from one or more corresponding uploaded images of the scene, and display and/or store the 
calculated information at the client workstation. Due to the centralized server configuration, 
more than one client may obtain position and/or size information regarding the same scene or 
group of scenes. In particular, according to one aspect of this embodiment, one or more 
images that are uploaded to a server may be archived at the server such that they are globally 
accessible to a number of designated users for one or more calculated measurements. 
Alternatively, according to another aspect, uploaded images may be archived such that they 
are only accessible to particular users. 

According to yet another embodiment of the invention related to network 
implementation of image metrology methods and apparatus, one or more images for 
processing are maintained at a client workstation, and the client downloads the appropriate 
image metrology algorithms from the server for one-time use as needed to locally process the 
images. In this aspect, a security advantage is provided for the client, as it is unnecessary to 
upload images over the network for processing by one or more servers. 

Following below are more detailed descriptions of various concepts related to, and 
embodiments of, image metrology methods and apparatus according to the present invention. 
It should be appreciated that various aspects of the invention as introduced above and 
discussed in greater detail below may be implemented in any of numerous ways, as the 
invention is not limited to any particular manner of implementation. Examples of specific 
implementations and applications are provided for illustrative purposes only. 
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B. Image Metrology Using A Single Image 

As discussed above, various embodiments of the invention are directed to manual or 
automatic image metrology methods and apparatus using a single image of a scene of interest. 
For these embodiments, Applicants have recognized that by considering certain types of 
scenes, for example, scenes that include essentially planar surfaces having known spatial 
relationships with one another, position and/or size information associated with objects of 
interest in the scene may be determined with respect to one or more of the planar surfaces 
from a single image of the scene. 

In particular, as shown for example in Fig. 5, Applicants have recognized that a variety 
of scenes including man-made or "built" spaces particularly lend themselves to image 
metrology using a single image of the scene, as typically such built spaces include a number 
of planar surfaces often at essentially right angles to one another (e.g., walls, floors, ceilings, 
etc.). For purposes of this disclosure, the term "built space" generally refers to any scene that 
includes at least one essentially planar man-made surface, and more specifically to any scene 
that includes at least two essentially planar man-made surfaces at essentially right angles to 
one another. More generally, the term "planar space" as used herein refers to any scene, 
whether naturally occurring or man-made, that includes at least one essentially planar surface, 
and more specifically to any scene, whether naturally occurring or man-made, that includes at 
least two essentially planar surfaces having a known spatial relationship to one another. 
Accordingly, as illustrated in Fig. 5, the portion of a room (in a home, office, or the like) 
included in the scene 20 may be considered as a built or planar space. 

As discussed above in connection with conventional photogrammetry techniques, 
often the exterior orientation of a particular camera relative to a scene of interest, as well as 
other camera calibration information, may be unknown a priori but may be determined, for 
example, in a resection process. According to one embodiment of the invention, at least the 
exterior orientation of a camera is determined using a number of reference objects that are 
located in a single plane, or "reference plane," of the scene. For example, in the scene 20 



shown in Fig. 5, the rear wall of the room (including the door, and on which a family portrait 
34 hangs) may be designated as a reference plane 21 for the scene 20. According to one 
aspect of this embodiment, the reference plane may be used to establish the reference 
coordinate system 74 for the scene; for example, as shown in Fig. 5, the reference plane 21 
(i.e., the rear wall) serves as an x-y plane for the reference coordinate system 74, as indicated 
by the x r and y r axes, with the z r axis of the reference coordinate system 74 perpendicular to 
the reference plane 21 and intersecting the x r andy r axes at the reference origin 56. The 
location of the reference origin 56 may be selected arbitrarily in the reference plane 21, as 
discussed further below in connection with Fig. 6. 

In one aspect of this embodiment, once at least the camera exterior orientation is 
determined with respect to the reference plane 21 (and, hence, the reference coordinate system 
74) of the scene 20 in Fig. 5, and given that at least the camera principle distance and perhaps 
other camera model parameters are known or reasonably estimated a priori (or also 
determined, for example, in a resection process), the coordinates of any point of interest in the 
reference plane 21 (e.g., corners of the door or family portrait, points along the backboard of 
the sofa, etc.) may be determined with respect to the reference coordinate system 74 from a 
single image of the scene 20, based on Eq. (11) above. This is possible because there are only 
two unknown (x- and y-) coordinates in the reference coordinate system 74 for points of 
interest in the reference plane 21 ; in particular, it should be appreciated that the z-coordinate 
in the reference coordinate system 74 of all points of interest in the reference plane 21, as 
defined, is equal to zero. Accordingly, the system of two collinearity equations represented 
by Eq. (11) may be solved as a system of two equations in two unknowns, using the two (x- 
andj>-) image coordinates of a single corresponding image point (i.e., from a single image) of 
a point of interest in the reference plane of the scene. In contrast, in a conventional 
intersection process as discussed above, generally all three coordinates of a point of interest in 
the scene are unknown; as a result, at least two corresponding image points (i.e., from two 
different images) of the point of interest are required to generate a system of four collinearity 
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equations in three unknowns to provide for a closed-form solution to Eq. (1 1) for the 

coordinates of the point of interest. 

It should be appreciated that the toee-dimensional coordinates in the reference 
coordinate system 74 of points of interest in the planar space shown in Fig. 5 may be 
determined from a single image of the scene 20 even if such points are located in various 
p,anes other than the designated reference plane 21 . In particular, any plane having a known 
(or determinable) spatial relationship to the reference plane 21 may serve as a •'measurement 
ptane » For example, in Fig. 5, the side wah (including the window and against winch the 
table with the vase is placed) and the floor of the room have a known or determinable spaUal 
relationship to the reference plane 21 (i.e., they are assumed to be a, essentially right angles 
with the reference plane 2 1); hence, the side wall may serve as a firs, measurement plane 23 
aud the floor may serve as a second measurement plane 25 in which coordinates of points of 
interest may be determined with respect to the reference coordinate system 74. 

For example, if Wo points 27A and 27B are identified in Fig. 5 a, the intersection of 
the measurement plane 23 and the reference plane 21, the location and orientation of the 
measurement plane 23 with respect to the reference coordinate system 74 may be determmed. 
in particular, the spatial relationship between the measurement plane 23 and the reference 
coordinate system 74 shown in Fig. 5 involves a 90 degree yaw rotation about the y, axts, and 
a translation along one or more of the *„ v„ and * axes of the reference coordinate system, as 
shown in Fig. 5 by the translation vector 55 (-/•„.). In one aspect, this translation vector may 
be ascertained from the coordinates of the points 27A and 27B as determined in the reference 
ptare 21 , as discussed further below. I. should be appreciated tat the foregoing is merely one 
example of how to link a measurement plane to a reference plane, and that other procedures 
for establishing such a relationship are suitable according to other embodiments of the 
invention. 

For purposes of illustration, Fig. 5 shows aset of measurement coordinate axes 57 
(i.e., an *„ axis and a v. axis) for the measurement plane 23. It should be appreciated mat an 
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origin 27C of the measurement coordinate axes 57 may be arbitrarily selected as any 
convenient point in the measurement plane 23 having known coordinates in the reference 
coordinate system 74 (e.g., one of the points 27A or 27B at the junction of the measurement 
and reference planes, other points along the measurement plane 23 having a known spatial 
relationship to one of the points 27A or 27B, etc.). It should also be appreciated that the y m 
axis of the measurement coordinate axes 57 shown in Fig. 5 is parallel to the y, axis of the 
reference coordinate system 74, and that the *, axis of the measurement coordinate axes 57 is 
parallel to the z, axis of the reference coordinate system 74. 

Once the spatial relationship between the measurement plane 23 and the reference 
plane 21 is known, and the camera exterior orientation relative to the reference plane 21 is 
known, the camera exterior orientation relative to the measurement plane 23 may be easily 
determined. For example, using the notation of Bq. (5), a coordinate system transformation 
»T from the reference coordinate system 74 to the measurement plane 23 may be derived 
blsed on the known translation vector 55 CP 0 , ) and a rotation matrix Z R that describes the 
coordinate axes rotation from the reference coordinate system to the measurement plane. In 
particular, in the example discussed above in connection with Fig. 5, the rotation matrix *R 
describes the 90 degree yaw rotation between the measurement plane and the reference plane. 
However, it should be appreciated that, in general, the measurement plane may have any 
arbitrary known spatial relationship to the reference plane, involving a rotation about one or 

more of three coordinate system axes. 

Ones the coordinate system transformation V is derived, the exterior orientation of 
the camera with respect to the measurement plane, based on the exterior orientation of the 
camera originally derived with respect to the reference plane, is represented in the 
transformation 
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Subsequently, the coordinates along the measurement coordinate axes 57 of any points of 
interest in the measurement plane 23 (e.g., corners of the window) may be determined from a 
single image of the scene 20, based on Eq. (1 1) as discussed above, by substituting [T in Eq. 
(1 1) with V of Eq. (17) to give coordinates of a point in the measurement plane from the 
image coordinates of the point as it appears in the single image. Again, it should be 
appreciated that closed-form solutions to Eq. (1 1) adapted in this manner are possible because 
there are only two unknown (*- and,-) coordinates for points of interest in the measurement 
plane 23, as the .-coordinate for such points is equal to zero by definition. Accordingly, the 
system of two collinearity equations represented by Eq. (1 1) adapted using Eq. (17) may be 
solved as a system of two equations in two unknowns. 

The determined coordinates with respect to the measurement coordinate axes 57 of 
points of interest in the measurement plane 23 may be subsequently converted to coordinates 
in the reference coordinate system 74 by applying an inverse transformation 'J , again based 
on the relationship between the reference origin 56 and the selected origin 27C of the 
measurement coordinate axes 57 given by the translation vector 55 and any coordinate axis 
rotations (e.g., a 90 degree yaw rotation). In particular, determined coordinates along the x m 
axis of the measurement coordinate axes 57 may be converted to coordinates along the z r axis 
of the reference coordinate system 74, and determined coordinates along the y m axis of the 
measurement coordinate axes 57 may be converted to coordinates along the y r axis of the 
reference coordinate system 74 by applying the transformation 'J. Additionally, it should be 
appreciated that all points in the measurement plane 23 shown in Fig. 5 have a same x- 
coordinate in the reference coordinate system 74. Accordingly, the three-dimensional 
coordinates in the reference coordinate system 74 of points of interest in the measurement 
plane 23 may be determined from a single image of the scene 20. 

Although one aspect of image metrology methods and apparatus according to the 
invention for processing a single image of a scene is discussed above using an example of a 
built space including planes intersecting at essentially right angles, it should be appreciated 
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that the invention is not limited in this respect. In particular, in various embodiments, one or 
more measurement planes in a planar space may be positioned and oriented in a known 
manner at other than right angles with respect to a particular reference plane. It should be 
appreciated that as long as the relationship between a given measurement plane and a 
reference plane is known, the camera exterior orientation with respect to the measurement 
plane may be determined, as discussed above in connection with Eq. (17). It should also be 
appreciated that, according to various embodiments, one or more points in a scene that 
establish a relationship between one or more measurement planes and a reference plane (e.g., 
the points 27 A and 27B shown in Fig. 5 at the intersection of two walls respectively defining 
the measurement plane 23 and the reference plane 21) may be manually identified in an 
image, or may be designated in a scene, for example, by one or more stand-alone robust 
fiducial marks (RFIDs) that facilitate automatic detection of such points in the image of the 
scene. In one aspect, each RFID that is used to identify relationships between one or more 
measurement planes and a reference plane may have one or more physical attributes that 
enable the RFID to be uniquely and automatically identified in an image. In another aspect, a 
number of such RFIDs may be formed on self-adhesive substrates that may be easily affixed 
to appropriate points in the scene to establish the desired relationships. 

Once the relationship between one or more measurement planes and a reference plane 
is known, three-dimensional coordinates in a reference coordinate system for the scene for 
points of interest in one or more measurement planes (as well as for points of interest in one 
or more reference planes) subsequently may be determined based on an appropriately adapted 
version of Eq. (11), as discussed above. The foregoing concepts related to coordinate system 
transformations between an arbitrary measurement plane and the reference plane are discussed 
in greater detail below in Section L of the Detailed Description. 

Additionally, it should be appreciated that in various embodiments of the invention 
related to image metrology methods and apparatus using single (or multiple) images of a 
scene, a variety of position and/or size information associated with objects of interest in the 
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scene may be derived based on three-dimensional eoordina.es of one or more points rn the 
scene with respect to a reference coordinate system for the scene. For example, a physical 
disuse between two points in the scene may be derived from the respectively determnred 
three-dimensional coordinates of each point based on fundamental geometric pnncp.es. 
From the foregoing, it should be appreciated that by ascribing a number of points to an object 
of interest, relative position and/or size information for a wide variety of objects may he 
determined based on the relative location in three dimensions of such points, and dishes 
between points that identify certain features of an object. 

C Exemplary Image Metrology Apparatus 

Fig 6 is a diagram illustrating an example of an image metrology apparatus according 
,0 one embodiment of the invention. In particular, Fig. 6 illustrates one example of an image 
metrology apparatus suitable for processing either a single image or multiple images of a 
scene to determine position and/or size information associated with objects of interest in the 

scene. 

in the embodiment of Fig. 6, the scene of interest 20A is shown, for example, as a 
portion of a room of some buih space (e.g., a home or an office), similar to tha, shown in F.g. 
5 ,n particular, tire scene 20A of Fig. 6 shows an essentially normal (i.e., "head-on") vtew of 
the rear wall of the scene 20 iUustrated in Fig. 5, which includes the door, tire family portrar. 
34 and tire sofa. Fig. 6 also shows tha, the scene 20A includes a reference target 1 20A tha, ,s 
piaced in the scene (e.g., also hanging on tire rear wall of the room). As discussed further 
below in connection with Fig. 8, known reference information associated with tire reference 
target 120A, as well as information derived from an image of the reference targe,, m par. 
facilitates a determination of position and/or size information associa.ed witir objects of 

interest in the scene. 

According to one aspect of me embodiment of Fig. 6, the reference target 120A 
establishes the reference plane 21 for the scene, and more specifically establishes me 
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reference coordinate system 74 for the scene, as indicated schematically in Fig. 6 by the x r and 
y r axes in the plane of the reference target, and the reference origin 56 (the z T axis of the 
reference coordinate system 74 is directed out of, and orthogonal to, the plane of the reference 
target 120A). It should be appreciated that while the x r and y r axes as well as the reference 
origin 56 are shown in Fig. 6 for purposes of illustration, these axes and origin do not 
necessarily actually appear^ se on the reference target 120A (although they may, according 
to some embodiments of the invention). 

As illustrated in Fig. 6, a camera 22 is used to obtain an image 20B of the scene 20A, 
which includes an image 120B of the reference target 120A that is placed in the scene. As 
discussed above, the term "camera" as used herein refers generally to any of a variety of 
image recording devices suitable for purposes of the present invention, including, but not 
limited to, metric or non-metric cameras, film or digital cameras, video cameras, digital 
scanners, and the like. According to one aspect of the embodiment of Fig. 6, the camera 22 
may represent one or more devices that are used to obtain a digital image of the scene, such as 
a digital camera, or the combination of a film camera that generates a photograph and a digital 
scanner that scans the photograph to generate a digital image of the photograph. In the latter 
case, according to one aspect, the combination of the film camera and the digital scanner may 
be considered as a hypothetical single image recording device represented by the camera 22 in 
Fig. 6. In general, it should be appreciated that the invention is not limited to use with any 
one particular type of image recording device, and that different types and/or combinations of 
image recording devices may be suitable for use in various embodiments of the invention. 

The camera 22 shown in Fig. 6 is associated with a camera coordinate system 76, 
represented schematically by the axes x c , y c , and z c , and a camera origin 66 (e.g., a nodal point 
of a lens or lens system of the camera), as discussed above in connection with Fig. 1. An 
optical axis 82 of the camera 22 lies along the * axis of the camera coordinate system 76. 
According to one aspect of this embodiment, the camera 22 may have an arbitrary spatial 
relationship to the scene 20A; in particular, the camera exterior orientation (i.e., the position 
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and orientation of the camera coordinate system 76 with respect to the reference coordinate 

system 74) may be unknown a priori. 

Fig. 6 also shows that the camera 22 has an image plane 24 on which the image 20B of 
the scene 20A is formed. As discussed above, the camera 22 may be associated with a 
particular camera model (e.g., including various interior orientation and lens distortion 
parameters) that describes the manner in which the scene 20A is projected onto the image 
plane 24 of the camera to form the image 20B. As discussed above, the exterior orientation of 
the camera, as well as the various parameters constituting the camera model, collectively are 
referred to in general as camera calibration information. 

According to one embodiment of the invention, the image metrology apparatus shown 
in Fig. 6 comprises an image metrology processor 36 to receive the image 20B of the scene 
20A. According to some embodiments, the apparatus also may include a display 38 (e.g., a 
CRT device), coupled to the image metrology processor 36, to display a displayed image 20C 
of the image 20B (including a displayed image 120C of the reference target 120A). 
Additionally, the apparatus shown in Fig. 6 may include one or more user interfaces, shown 
for example as a mouse 40A and a keyboard 40B, each coupled to the image metrology 
processor 36. The user interfaces 40A and/or 40B allow a user to select (e.g., via point and 
click using a mouse, or cursor movement) various features of interest that appear in the 
displayed image 20C (e.g., the two points 26B and 28B which correspond to actual points 
26A and 28A, respectively, in the scene 20A). It should be appreciated that the invention is 
not limited to the user interfaces illustrated in Fig. 6; in particular, other types and/or 
additional user interfaces not explicitly shown in Fig. 6 (e.g., a touch sensitive display screen, 
various cursor controllers implemented on the keyboard 40B, etc.) may be suitable in other 
embodiments of the invention to allow a user to select one or more features of interest in the 



scene. 



According to one embodiment, the image metrology processor 36 shown in Fig. 6 
determines, from the single image 20B, position and/or size information associated with one 
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or more objects of interest in the scene 20A, based at least in part on the reference information 
associated with the reference target 120A, and information derived from the image 120B of 
the reference target 120A. In this respect, it should be appreciated that the image 20B 
generally includes a variety of other image content of interest from the scene in addition to the 
image 120B of the reference target. According to one aspect of this embodiment, the image 
metrology processor 36 also controls the display 38 so as to provide one or more indications 
of the determined position and/or size information to the user. 

For example, according to one aspect of this embodiment, as illustrated in Fig. 6, the 
image metrology processor 36 may calculate a physical (i.e., actual) distance between any two 
points in the scene 20A that lie in a same plane as the reference target 120A. Such points 
generally may be associated, for example, with an object of interest having one or more 
surfaces in the same plane as the reference target 120A (e.g., the family portrait 34 shown in 
Fig. 6). In particular, as shown in Fig. 6, a user may indicate (e.g., using one of the user 
interfaces 40A and 40B) the points of interest 26B and 28B in the displayed image 20C, 
which points correspond to the points 26A and 28A at two respective corners of the family 
portrait 34 in the scene 20A, between which a measurement of a physical distance 30 is 
desired. Alternatively, according to another embodiment of the invention, one or more stand- 
alone robust fiducial marks (RFIDs) may be placed in the scene to facilitate automatic 
detection of points of interest for which position and/or size information is desired. For 
example, an RFID may placed in the scene at each of the points 26A and 28A, and these 
RFIDs appearing in the image 20B of the scene may be automatically detected in the image to 

indicate the points of interest. 

In this aspect of the embodiment shown in Fig. 6, the processor 36 calculates the 
distance 30 and controls the display 38 so as to display one or more indications 42 of the 
calculated distance. For example, an indication 42 of the calculated distance 30 is shown in 
Fig. 6 by the double-headed arrow and proximate alphanumeric characters "1 m." (i.e., one 
meter), which is superimposed on the displayed image 20C near the selected points 26B and 
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28B. It should be appreciated, however, that the invention is not limited in this respect, as 
other methods for providing one or more indications of calculated physical distance 
measurements, or various other position and/or size information of objects of interest in the 
scene, may be suitable in other embodiments (e.g., one or more audible indications, a hard- 
copy printout of the displayed image with one or more indications superimposed thereon, 
etc.). 

According to another aspect of the exemplary image metrology apparatus shown in 
Fig. 6, a user may select (e.g., via one or more user interfaces) a number of different pairs of 
points in the displayed image 20C from time to time (or alternatively, a number of different 
pairs of points may be uniquely and automatically identified by placing a number of stand- 
alone RFIDs in the scene at desired locations), for which physical distances between 
corresponding pairs of points in the reference plane 21 of the scene 20A are calculated. As 
discussed above, indications of the calculated distances subsequently may be indicated to the 
user in a variety of manners (e.g., displayed / superimposed on the displayed image 20C, 
printed out, etc.). 

In the embodiment of Fig. 6, it should be appreciated that the camera 22 need not be 
coupled to the image metrology processor 36 at all times. In particular, while the processor 
may receive the image 20B shortly after the image is obtained, alternatively the processor 36 
may receive the image 20B of the scene 20A at any time, from a variety of sources. For 
example, the image 20B may be obtained by a digital camera, and stored in either camera 
memory or downloaded to some other memory (e.g., a personal computer memory) for a 
period of time. Subsequently, the stored image may be downloaded to the image metrology 
processor 36 for processing at any time. Alternatively, the image 20B may be recorded using 
a film camera from which a print (i.e., photograph) of the image is made. The print of the 
image 20B may then be scanned by a digital scanner (not shown specifically in Fig. 5), and 
the scanned print of the image may be directly downloaded to the processor 36 or stored in 
scanner memory or other memory for a period of time for subsequent downloading to the 
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processor 36. 

From .he foregoing, as discussed above, it should be appreciated that a variety of 
image recording devices (e.g., digit*, or film cameras, digital scanners, video recorders, etc.) 
m ay be used from time ,0 time to acquire one or more images of scenes suitable for image 
metrology processing according to various embodiments of the present invention. In any 
case, according to one aspect of the embodiment of Fig. 6, a user places the reference targe. 
12 0A ^particular plane of interest u> es^blish the reference piane 21 for *e scene, obtams 
an image of me scene including the reference target 120A, and downloads fte image a. some 
convenient time to the image metrology processor 36 to obtain position and/or size 
information associated with objects of interest in the reference plane of the scene. 

D Exemplary Image Metrology Applications 

The exemplary image metrology appamtus of Fig. 6, as well as image metrology 
apparatus according to other embodiments of the invention, generally are suitable for a wrde 
variety of applications, including those in which users desire measurement, of indoor or 
outdoor built (or, in genera., planar) spaces. For examp.e, contractors or architects may use 
an image metro.ogy apparatus of the inventionfor project design, remodehng and es.imat.on 
of work on built (or to-be-built) spaces. Similarly, building appraisers and insurance 
estimators may derive useful measurement-related information using an image metro.ogy 
apparatus of the invention. Likewise, realtors may present various building floor plans ,0 
potential buyers who can compare dimensions of spaces and/or ascertain if various 
furnishings will fit in spaces, and interior designers can demonstrate interior design tdeas to 

potential customers. 

Additionally, law enforcement agents may use an image metrology apparatus 
according to the invention for a variety of forensic investigations in which spatial 
relationships a, a crime scene may be important. In crime scene anaiysis, vaiuable evidence 
often may be lost if details of the scene are no, observed and/or recorded immediate.,. An 
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image metrology apparatus according to the invention enables law enforcement agents to 
obtain images of a crime scene easily and quickly, under perhaps urgent and/or emergency 
circumstances, and then later download the images for subsequent processing to obtain a 
variety of position and/or size information associated with objects of interest in the scene. 

It should be appreciated that various embodiments of the invention as discussed herein 
may be suitable for one or more of the foregoing applications, and that the foregoing 
applications are not limited to the image metrology apparatus discussed above in connection 
with Fig. 6. Likewise, it should be appreciated 'that image metrology methods and apparatus 
according to various embodiments of the present invention are not limited to the foregoing 
applications, and that such exemplary applications are discussed herein for purposes of 
illustration only. 

K Exemplary Network Implementations of Image Metrology Methods and 
Apparatus 

Fig. 7 is a diagram illustrating an image metrology apparatus according to another 
embodiment of the invention. The apparatus of Fig. 7 is configured as a "client-server" image 
metrology system suitable for implementation over a local-area network or a wide-area 
network, such as the Internet. In the system of Fig. 7, one or more image metrology servers 
36A, similar to the image metrology processor 36 of Fig. 6, are coupled to a network 46, 
which may be a local-area or wide-area network (e.g., the Internet). An image metrology 
server 36A provides image metrology processing services to a number of users (i.e., clients) at 
client workstations, illustrated in Fig. 7 as two PC-based workstations 50A and SOB, that are 
also coupled to the network 46. While Fig. 7 shows only two client workstations 50A and 
SOB, it should be appreciated that any number of client workstations may be coupled to the 
network 46 to download information from, and upload information to, one or more image 

metrology servers 3 6 A. 

Fig. 7 shows that each client workstation 50A and SOB may include a workstation 
processor 44, (e.g., a personal computer), one or more user interfaces (e.g., a mouse 40A and 
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a keyboard 40B), and a display 38. Fig. 7 also shows that one or more cameras 22 may be 
coupled to each workstation processor 44 from time to time, to download recorded images 
locally at the client workstations. For example, Fig. 7 shows a scanner coupled to the 
workstation 50A and a digital camera coupled to the workstation 50B. Images recorded by 
either of these recording devices (or other types of recording devices) may be downloaded to 
any of the workstation processors 44 at any time, as discussed above in connection with Fig. 
6. It should be appreciated that one or more same or different types of cameras 22 may be 
coupled to any of the client workstations from time to time, and that the particular 
arrangement of client workstations and image recording devices shown in Fig. 7 is for 
purposes of illustration only. Additionally, for purposes of the present discussion, it is 
understood that each workstation processor 44 is operated using one or more appropriate 
conventional software programs for routine acquisition, storage, and/or display of various 
information (e.g., images recorded using various recording devices). 

In the embodiment of an image metrology apparatus shown in Fig. 7, it should also be 
appreciated for purposes of the present discussion that each client workstation 44 coupled to 
the network 46 is operated using one or more appropriate conventional client software 
programs that facilitate the transfer of information across the network 46. Similarly, it is 
understood that the image metrology server 36A is operated using one or more appropriate 
conventional server software programs that facilitate the transfer of information across the 
network 46. Accordingly, in embodiments of the invention discussed further below, the 
image metrology server 36A shown in Fig. 7 and the image metrology processor 36 shown in 
Fig. 6 are described similarly in terms of those components and functions specifically related 
to image metrology that are common to both the server 36A and the processor 36. In 
particular, in embodiments discussed further bek>w, image metrology concepts and features 
discussed in connection with the image metrology processor 36 of Fig. 6 similarly relate and 
apply to the image metrology server 36A of Fig. 7. 

According to one aspect of the network-based image metrology apparatus shown in 
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Fig. 7, each of the client workstations 50A and 50B may upload image-related information to 
the image metrology server 36A at any time. Such image-related information may include, 
for example, the image of the scene itself (e.g., the image 20B from Fig. 6), as well as any 
points selected in the displayed image by the user (e.g., the points 26B and 28B in the 
displayed image 20C in Fig. 6) which indicate objects of interest for which position and/or 
size information is desired. In this aspect, the image metrology server 36A processes the 
uploaded information to determine the desired position and/or size information, after which 
the image metrology server downloads to one or more client workstations the desired 
information, which may be communicated to a user at the client workstations in a variety of 
manners (e.g., superimposed on the displayed image 20C). 

In yet another aspect of the network-based image metrology apparatus shown in Fig. 7, 
rather than uploading images from one or more client workstations to an image metrology 
server, images are maintained at client workstations and the appropriate image metrology 
algorithms are downloaded from the server to the clients for use as needed to locally process 
the images. In this aspect, a security advantage is provided for the client, as it is unnecessary 
to upload images over the network for processing by one or more image metrology servers. 

F. Exemplary Network-based Image Metrology Applications 
As with the image metrology apparatus of Fig. 6, various embodiments of the 
network-based image metrology apparatus shown in Fig. 7 generally are suitable for a wide 
variety of applications in which users require measurements of objects in a scene. However, 
unlike the apparatus of Fig. 6, in one embodiment the network-based apparatus of Fig. 7 may 
allow a number of geographically dispersed users to obtain measurements from a same image 
or group of images. 

For example, in one exemplary application of the network-based image metrology 
apparatus of Fig. 7, a realtor (or interior designer, for example) may obtain images of scenes 
in a number of different rooms throughout a number of different homes, and upload these 
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images (e.g., from their own client workstation) to the image metrology server 36A. The 
uploaded images may be stored in the server for any length of time. Interested buyers or 
customers may connect to the realtor's (or interior designer's) webpage via a client 
workstation, and from the webpage subsequently access the image metrology server 36A. 
From the uploaded and stored images of the homes, the interested buyers or customers may 
request image metrology processing of particular images to compare dimensions of various 
rooms or other spaces from home to home. In particular, interested buyers or customers may 
determine whether personal furnishings and other belongings, such as furniture and 
decorations, will fit in the various living spaces of the home. In this manner, potential buyers 
or customers can compare homes in a variety of geographically different locations from one 
convenient location, and locally display and/or print out various images of a number of rooms 
in different homes with selected measurements superimposed on the images. 

As discussed above, it should be appreciated that network implementations of image 
metrology methods and apparatus according to various embodiments of the present invention 
are not limited to the foregoing exemplary application, and that this application is discussed 
herein for purposes of illustration only. Additionally, as discussed above in connection with 
Fig. 7, it should be appreciated in the foregoing example that images alternatively may be 
maintained at client workstations, and the appropriate image metrology algorithms may be 
downloaded from the server (e.g., via a service provider's webpage) to the clients for use as 
needed to locally process the images and preserve security. 

G. Exemplary Reference Objects for Image Metrology Methods and Apparatus 
Accordihg to one embodiment of the invention as discussed above in connection with 
Figs. 5 and 6, the image metrology processor 36 shown in Fig. 6 first determines various 
camera calibration information associated with the camera 22 in order to ultimately determine 
position and/or size information associated with one or more objects of interest in the scene 
20A that appear in the image 20B obtained by the camera 22. For example, according to one 
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embodiment, the image metrology processor 36 determines at least the exterior orientation of 
the camera 22 (i.e., the position and orientation of the camera coordinate system 76 with 
respect to the reference coordinate system 74 for the scene 20A, as shown in Fig. 6). 

In one aspect of this embodiment, the image metrology processor 36 determines at 
least the camera exterior orientation using a resection process, as discussed above, based at 
least in part on reference information associated with reference objects in the scene, and 
information derived from respective images of the reference objects as they appear in an 
image of the scene. In other aspects, the image metrology processor 36 determines other 
camera calibration information (e.g., interior orientation and lens distortion parameters) in a 
similar manner. As discussed above, the term "reference information" generally refers to 
various information (e.g., position and/or orientation information) associated with one or more 
reference objects in a scene that is known a priori with respect to a reference coordinate 
system for the scene. 

In general, it should be appreciated that a variety of types, numbers, combinations and 
arrangements of reference objects may be included in a scene according to various 
embodiments of the invention. For example, various configurations of reference objects 
suitable for purposes of the invention include, but are not limited to, individual or "stand- 
alone" reference objects, groups of objects arranged in a particular manner to form one or 
more reference targets, various combinations and arrangements of stand-alone reference 
objects and/or reference targets, etc. The configuration of reference objects provided in 
different embodiments may depend, in part, upon the particular camera calibration 
information (e.g., the number of exterior orientation, interior orientation, and/or lens distortion 
parameters) that an image metrology method or apparatus of the invention needs to determine 
for a given application (which, in turn, may depend on a desired measurement accuracy). 
Additionally, according to some embodiments, particular types of reference objects may be 
provided in a scene depending, in part, on whether one or more reference objects are to be 
identified manually or automatically from an image of the scene, as discussed further below. 
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Gl. Exemplary Reference Targets 
In view of the foregoing, one embodiment of the present invention is directed to a 
reference target that, when placed in a scene of interest, facilitates a determination of various 
camera calibration information. In particular, Fig. 8 is a diagram showing an example of the 
reference target 120A that is placed in the scene 20A of Fig. 6, according to one embodiment 
of the invention. It should be appreciated however, as discussed above, that the invention is 
not limited to the particular example of the reference target 120A shown in Fig. 8, as 
numerous implementations of reference targets according to various embodiments of the 
invention (e.g., including different numbers, types, combinations and arrangements of 

reference objects) are possible. 

According to one aspect of the embodiment shown in Fig. 8, the reference target 120A 
is designed generally to be portable, so that it is easily transferable amongst different scenes 
and/or different locations in a given scene. For example, in one aspect, the reference target 
120A has an essentially rectangular shape and has dimensions on the order of 25 cm. In 
another aspect, the dimensions of the reference target 120A are selected for particular image 
metrology applications such that the reference target occupies on the order of 1 00 pixels by 
100 pixels in a digital image of the scene in which it is placed. It should be appreciated, 
however, that the invention is not limited in these respects, as reference targets according to 
other embodiments may have different shapes and sizes than those indicated above. 

In Fig. 8, the example of the reference target 120A has an essentially planar front (i.e., 
viewing) surface 121 , and includes a variety of reference objects that are observable on at 
least the front surface 121. In particular, Fig. 8 shows that the reference target 120A includes 
four fiducial marks 124A, 124B, 124C, and 124D, shown for example in Fig. 8 as asterisks. 
In one aspect, the fiducial marks 124A-124D are similar to control points, as discussed above 
in connection with various photogrammetry techniques (e.g., resection). Fig. 8 also shows 
that the reference target 120A includes a first orientation-dependent radiation source (ODR) 
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122A and a second ODR 122B. 

According to one aspect of the embodiment of the reference target 120A shown in Fig. 
8, the fiducial marks 124A-124D have known spatial relationships to each other. 
Additionally, each fiducial mark 124A-124D has a known spatial relationship to the ODRs 
122 A and 122B. Stated differently, each reference object of the reference target 120 A has a 
known spatial relationship to at least one point on the target, such that relative spatial 
information associated with each reference object of the target is known a priori. These 
various spatial relationships constitute at least some of the reference information associated 
with the reference target 120 A. Other types of reference information that may be associated 
with the reference target 120 A are discussed further below. 

In the embodiment of Fig. 8, each ODR 122 A and 122B emanates radiation having at 
least one detectable property, based on an orientation of the ODR, that is capable of being 
detected from an image of the reference target 120A (e.g., the image 120B shown in Fig. 6). 
According to one aspect of this embodiment, the ODRs 122A and 122B directly provide 
particular information in an image that is related to an orientation of the camera relative to the 
reference target 120A, so as to facilitate a determination of at least some of the camera 
exterior orientation parameters. According to another aspect, the ODRs 122 A and 122B 
directly provide particular information in an image that is related to a distance between the 
camera (e.g. the camera origin 66 shown in Fig. 6) and the reference target 120A. The 
foregoing and other aspects of ODRs in general are discussed in greater detail below, in 
Sections G2 and J of the Detailed Description. 

As illustrated in Fig. 8, each ODR 122A and 122B has an essentially rectangular shape 
defined by a primary axis that is parallel to a long side of the ODR, and a secondary axis, 
orthogonal to the primary axis, that is parallel to a short side of the ODR. In particular, in the 
exemplary reference target shown in Fig. 8, the ODR 122 A has a primary axis 130 and a 
secondary axis 132 that intersect at a first ODR reference point 125 A. Similarly, in Fig. 8, the 
ODR 122B has a secondary axis 138 and a primary axis which is coincident with the 
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secondary ax, 132 of the ODR 122A. Theses 138 and >32ofthe ODR 122B in,ersec,a,a 
second ODR reference poin, 125B. It should be appreciated «ha« the invention is not tatted 
to the ODRs 122A and 122B sharing one or more axes (as shown in Fig. 8 by the axis 1 32), 
and that the particular arrangement and gene., shape of «he ODRs shown in Fig. 8 ts for 
purposes of illustration only. In particular, according to other embodiments, the ODR 

may have a primary axis tha, does no. coincide wi,h the secondary axis 1 32 of the ODR 

122A. 

According to one aspect of the exempiary embodiment shown in Fig. 8, the ODRs 
122A and 122B are arranged in the reference target 120A such that their respective pnmary 
axes 130 and 132 are orthogonal to each outer and each parallel to a side of the reference 
^get However, it should be appreciated mat the invention is no. limited in mis respect, as 
various ODRs may be differently oriented (i.e., no, necessarily orthogonal to each outer) nr a 
referencetargethavinganessentiallyrec^gularoromershape.accordingtoomer 

embodiments. Arbitrary orientations of ODRs (e.g., ormogonal vs. non-orthogonal) mcluded 
in reference targets according to various embodiments of the invention are discussed ,n 
greater detail in Section L of the Detailed Description. 

According to another aspect of the exempiary embodiment shown in Fig. 8, the ODRs 
122A and 122B are arranged in me reference targe, 120A such tha, each of their respecUve 
secondary axes 132 and 138 passes through a common intersection point 140 of the reference 
targe, While Fig. 8 shows « he primary axis of the ODR 122B also passing through the 
common infection point 140 of the reference target .20A, it should be appreciated tha, me 
inV en.ion is not limited in this respect (i.e., the primary axis of the ODR 122B does no, 
necessarily pass through the common intersection coin, , 40 of me reference ta^e, 120A 
according to other embodiments of me invention), in particular, as discussed above, the 
coincidence of the primary axis of the ODR 122B and the secondary axis of the ODR 122A 
(such ma, the second ODR reference poin, 125B coincides with me common in,ersec„on 
poin, 140) is merely one design option implemented in the particular example shown m Ftg. 8. 
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In yet another aspect, the common intersection point 140 may coincide with a geometric 
center of the reference target, but again it should be appreciated that the invention is not 

limited in this respect. 

According to one embodiment of the invention, as shown in Fig. 8, the secondary axis 
138 of the ODR 122B serves as an axis of the reference target 120A, and the secondary axis 
132 of the ODR 122A serves as a* axis of the reference target. In one aspect of this 
embodiment, each fiducial mark 124A-124D shown in the target of Fig. 8 has a known spatial 
relationship to the common intersection point 140. In particular, each fiducial mark 124A- 
124D has known "target" coordinates with respect to the x, axis 138 and the* axis 132 of the 
reference target 120A. Likewise, the target coordinates of the first and second ODR reference 
points 125A and 125B are known with respect to the x, axis 138 and the * axis 132. 
Additionally, the physical dimensions of each of the ODRs 122A and 122B (e.g., length and 
width for essentially rectangular ODRs) are known by design. In this manner, a spatial 
position (and, in some instances, extent) of each reference object of the reference target 120A 
shown in Fig. 8 is known a priori with respect to the x, axis 138 and the* axis 132 of the 
reference target 120A. Again, this spatial information constitutes at least some of the 
reference information associated with the reference target 120A. 

With reference again to both Figs. 6 and 8, in one embodiment, the common 
intersection point 140 of the reference target 120A shown in Fig. 8 defines the reference 
origin 56 of the reference coordinate system 74 for the scene in which the reference target is 
placed. In one aspect of this embodiment, the x, axis 138 and the* axis 132 of the reference 
target lie in the reference plane 21 of the reference coordinate system 74, with a normal to the 
reference target that passes through the common intersection point 140 defining the z r axis of 
the reference coordinate system 74 (i.e., out of the plane of both Figs. 6 and 8). 

In particular, in one aspect of this embodiment, as shown in Fig. 6, the reference target 
120A may be placed in the scene such that the x, axis 138 and the* axis 132 of the reference 
target respectively correspond to the x r axis 50 and the * axis 52 of the reference coordinate 
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system 74 (i.e., the reference target axes essentially define the* axis 50 and the,, axis 52 of 
the reference coordinate system 74). Alternatively, in another aspect (no. shown in Ac 
figures), the x, and* axes of the reference targe, may lie in the reference plane 21, but the 
reference targe, may have a known "roll" rotation with respect to the * axis 50 and the y, axrs 
52 of the reference coordinate system 74; namely, the reference targe. 120A shown in F,g. 8 
may be rotated by a known amount about the normal to the target passing through the 
common intersection point 140 (i.e., about the * axis of the reference coordinate system 
shown in Fig. 6), such that the * and y, axes of the reference targe, arc no. respectively 
aligned with the * and* axes of tire reference coordinate system 74. Such a roll rotation of 
.he reference targe. 120A is discussed in greater detail in Section L of me Detailed 
Description. In ei.her of the above situations, however, in this embodiment the reference 
target 120A essentially defines the reference coordinate system 74 for the scene, either 
explicitly or by having a known roll rotation with respect to the reference plane 21. 

As discussed in greater detail further below in Sections G2 and J of tire Detailed 
Description, according to one embodiment the ODR 122A shown in Fig. 8 emanates 
orientation-dependent radiation 126A that varies as a function of a rotation 136 of the ODR 
122A about its secondary axis 132. Similarly, the ODR 122B in Fig. 8 emanates orientation- 
dependent radiation 126B tha, varies as a function of a rotation 134 of the ODR 122B about 

its secondary axis 1 3 8 . 

For purposes of providing an introductory explanation of the operation of the ODRs 
122A and 122B of the reference target 120A, Fig. 8 schematically illustrates each of the 
orientation dependent radiation 126A and 126B as a series of three oval-shaped radiation 
spots emanating from a respective observation surface 128A and 128B of the ODRs 122A and 
122B It should be appreciated, however, that the foregoing is merely one exemplary 
representation of the orientation dependent radiation 126A and 126B, and that the invention is 
not limited in this respect. With reference to the illustration of Fig. 8, according to one 
embodiment, the three radiation spots of each ODR collectively move along the primary axis 
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of the ODR (as indicated in Fig. 8 by the oppositely directed arrows on the observation 
surface of each ODR) as the ODR is rotated about its secondary axis. Hence, in this example, 
at least one detectable property of each of the orientation dependent radiation 126A and 126B 
is related to a position of one or more radiation spots (or, more generally, a spatial distribution 
of the orientation dependent radiation) along the primary axis on a respective observation 
surface 128A and 128B of the ODRs 122A and 122B. Again, it should be appreciated that the 
foregoing illustrates merely one example of orientation dependent radiation (and a detectable 
property thereof) that may be emanated by an ODR according to various embodiments of the 
invention, and that the invention is not limited to this particular example. 

Based on the general operation of the ODRs 122A and 122B as discussed above, in 
one aspect of the embodiment shown in Fig. 8, a "yaw" rotation 136 of the reference 
target 120A about its* axis 132 (i.e., the secondary axis of the ODR 122A) causes a variation 
of the orientation-dependent radiation 126A along the primary axis 130 of the ODR 122A 
(i.e., parallel to the x, axis 138). Similarly, a "pitch" rotation 134 of the reference target 120A 
about its Xt axis 138 (i.e., the secondary axis of the ODR 122B) causes a variation in the 
orientation-dependent radiation 126B along the primary axis 132 of the ODR 122B (i.e., along 
the* axis). In this manner, the ODRs 122A and 122B of the reference target 120A shown in 
Fig. 8 provide orientation information associated with the reference target in two orthogonal 
directions. According to one embodiment, by detecting the orientation-dependent radiation 
126A and 126B from an image 120B of the reference target 120A, the image metrology 
processor 36 shown in Fig. 6 can determine the pitch rotation 134 and the yaw rotation 136 of 
the reference target 120A. Examples of such a process are discussed in greater detail in 
Section L of the Detailed Description. 

According to one embodiment, the pitch rotation 134 and the yaw rotation 136 of the 
reference target 120A shown in Fig. 8 correspond to a particular "camera bearing" (i.e., 
viewing perspective) from which the reference target is viewed. As discussed further below 
and in Section L of the Detailed Description, the camera bearing is related to at least some of 
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the camera exterior orientation parameters. Accordingly, by directly providing information 
with respect to the camera bearing in an image of the scene, in one aspect the reference target 
120A advantageously facilitates a determination of the exterior orientation of the camera (as 
well as other camera calibration information). In particular, a reference target according to 
various embodiments of the invention generally may include automatic detection means for 
facilitating an automatic detection of the reference target in an image of the reference target 
obtained by a camera (some examples of such automatic detection means are discussed below 
in Section G3 of the Detailed Description), and bearing determination means for facilitating a 
determination of one or more of a position and at least one orientation angle of the reference 
target with respect to the camera (i.e., at least some of the exterior orientation parameters). In 
one aspect of this embodiment, one or more ODRs may constitute the bearing determination 
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Fig. 9 is a diagram illustrating the concept of camera bearing, according to one 
embodiment of the invention. In particular, Fig. 9 shows the camera 22 of Fig. 6 relative to 
the reference target 120A that is placed in the scene 20A. In the example of Fig. 9, for 
purposes of illustration, the reference target 1 20 A is shown as placed in the scene such that its 
x t axis 138 and its y t axis 132 respectively correspond to the x r axis 50 and the y r axis 52 of the 
reference coordinate system 74 (i.e., there is no roll of the reference target 120A with respect 
to the reference plane 21 of the reference coordinate system 74). Additionally, in Fig. 9, the 
common intersection point 140 of the reference target coincides with the reference origin 56, 
and the z r axis 54 of the reference coordinate system 74 passes through the common 
intersection point 140 normal to the reference target 120A. 

For purposes of this disclosure, the term "camera bearing" generally is defined in 
terms of an azimuth angle a 2 and an elevation angle y 2 of a camera bearing vector with respect 
to a reference coordinate system for an object being imaged by the camera. In particular, with 
reference to Fig. 9, in one embodiment, the camera bearing refers to an azimuth angle a 2 and 
an elevation angle y 2 of a camera bearing vector 78, with respect to the reference coordinate 
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system 74. As shown in Fig. 9 (and also in Fig. 1), the camera bearing vector 78 connects the 
origin 66 of the camera coordinate system 76 (e.g., a nodal point of the camera lens system) 
and the origin 56 of the reference coordinate system 74 (e.g., the common intersection point 
140 of the reference target 120A). In other embodiments, the camera bearing vector may 
connect the origin 66 to a reference point of a particular ODR. 

Fig. 9 also shows a projection 78' (in the x r - z r plane of the reference coordinate 
system 74) of the camera bearing vector 78, for purposes of indicating the azimuth angle « 2 
and the elevation angle y 2 of the camera bearing vector 78; in particular, the azimuth angle a, 
is the angle between the camera bearing vector 78 and the y r - z r plane of the reference 
coordinate system 74, and the elevation angle y 2 is the angle between the camera bearing 
vector 78 and the x r - z r plane of the reference coordinate system. 

From Fig. 9, it may be appreciated that the pitch rotation 134 and the yaw rotation 136 
indicated in Figs. 8 and 9 for the reference target 120A correspond respectively to the 
elevation angle y 2 and the azimuth angle a 2 of the camera bearing vector 78. For example, if 
the reference target 120A shown in Fig. 9 were originally oriented such that the normal to the 
reference target passing through the common intersection point 140 coincided with the camera 
bearing vector 78, the target would have to be rotated by y 2 degrees about its *, axis (i.e., a 
pitch rotation of y 2 degrees) and by a 2 degrees about its;;, axis (i.e., a yaw rotation of a 2 
degrees) to correspond to the orientation shown in Fig. 9. Accordingly, from the discussion 
above regarding the operation of the ODRs 122A and 122B with respect to pitch and yaw 
rotations of the reference target 120A, it may be appreciated from Fig. 9 that the ODR 122A 
facilitates a determination of the azimuth angle a 2 of the camera bearing vector 78, while the 
ODR 122B facilitates a determination of the elevation angle y 2 of the camera bearing vector. 
Stated differently, each of the respective oblique viewing angles of the ODRs 122A and 122B 
(i.e., rotations about their respective secondary axes) constitutes an element of the camera 
bearing. 

In view of the foregoing, it should be appreciated that other types of reference 
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information associated with reference objects of the reference target 120A shown in Fig. 8 
that may be known a priori (i.e., in addition to the relative spatial information of reference 
objects with respect to the x, and y, axes of the reference target, as discussed above) relates 
particularly to the ODRs 122A and 122B. In one aspect, such reference information 
associated with the ODRs 122A and 122B facilitates an accurate determination of the camera 
bearing based on the detected orientation-dependent radiation 126A and 126B. 

More specifically, in one embodiment, a particular characteristic of the detectable 
property of the orientation-dependent radiation 126A and 126B respectively emanated from 
the ODRs 122A and 122B as the reference target 120A is viewed "head-on" (i.e., the 
reference target is viewed along the normal to the target at the common intersection point 140) 
may be known a priori and constitute part of the reference information for the target 120A. 
For instance, as illustrated in the example of Fig. 8, a particular position along an ODR 
primary axis of one or more of the oval-shaped radiation spots representing the orientation- 
dependent radiation 126A and 126B, as the reference target is viewed along the normal, may 
be known a priori for each ODR and constitute part of the reference information for the target 
120A. In one aspect, this type of reference information establishes baseline data for a "normal 
camera bearing" to the reference target (e.g., corresponding to a camera bearing having an 
azimuth angle a 2 of 0 degrees and an elevation angle y 2 of 0 degrees, or no pitch and yaw 
rotation of the reference target). 

Furthermore, a rate of change in the characteristic of the detectable property of the 
orientation-dependent radiation 126A and 126B, as a function of rotating a given ODR about 
its secondary axis (i.e., a "sensitivity" of the ODR to rotation), may be known a priori for 
each ODR and constitute part of the reference information for the target 120A. For instance, 
as illustrated in the example of Fig. 8 (and discussed in detail in Section J of the Detailed 
Description), how much the position of one or more radiation spots representing the 
orientation-dependent radiation moves along the primary axis of an ODR for a particular 
rotation of the ODR about its secondary axis may be known a priori for each ODR and 
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constitute part of the reference information for the target 120A. 

In sum, examples of reference information that may be known a priori in connection 
with reference objects of the reference target 120A shown in Fig. 8 include, bu, are no. 
necessarily limited to, a size of the reference target !20A (i.e. physical dimensions of the 
taIg e,) the coordinates of the fiducial marks 124A-124D and the ODR reference points 125A 
and 125B with respect to the, and* axes of the reference target, the physical dimensions 
(e g length and width) of each of the ODRs 122A and 122B, respective basehne 
characteristics of one or more detectable properties of the orientation-dependent radiauon 
emanated from each ODR a, normal or "head-on" viewing of the targe,, and respective 
sensitivities of each ODR to rotation. Based on the foregoing, it should be appreciated that 
the various reference information associated with a given reference targe, may be untque to 
«ha, targe, (i.e., "target-specific" reference information), based in part on the type, number, 
m d particular combination and arrangement of reference objects included in the target. 

As discussed above (and in greater de<ail mrther below in Section L of the Detatled 
Description), according ,o one embodimen, of the invention, ,he image metrology processor 
36 of Fig 6 uses target-specific reference information associated with reference objects of a 
par.icu.ar reference target, along with information derived from an image of ,he reference 
target (e.g., «he image 120B in Fig. 6), to determine various camera calibration informal. 
,„ one aspect of this embodiment, such .argCspecific reference information may be manually 
input to the image metrology processor 36 by a user (e.g., via one or more user interfaces 40A 
and 40B) Once such reference information is input to the image metrology processor for a 
particular reference «arge,, that reference target may be used repeatedly in different scenes for 
which one or more images are downloaded to the processor for various image metrology 
purposes. 

!n another aspect, target-specific reference information for a particular reference targe, 
may be mainlined on a s.orage medium (e.g., floppy disk, CD-ROM) and downloaded ,o the 
image metrology processor a, any convenient time. For examp.e, according to one 
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embodiment a storage medium storing ,arge,-specific reference information for a pabular 

be portabiy used with different image metrology processors by down.oadi„g ,0 the processor 
me information stored on the medium. In another embodiment, target-specific informal for 
a particular reference target may be associated with a unique seria, number, so that a g.ve„ 
image metroiogy pressor can download and/or store, and easily identity, the 
information for a number of different reference targets mat are catalogued by unique sena, 
numbers, .n yet another embodiment, a particular reference target and image metrology 
processor may be packaged as a system, wherein the target-specific information for the 
reference target is initial* maintained in the image metroiogy processor's semi-permanent or 
permanent memory (e.g., ROM, EEPROM). From the foregoing, it should be appreciated that 
a wide vane* of methods for making reference information available to an image metrology 
processor are suitable according to various embodiments of the invention, and that the 
invention is not limited to the foregoing examples. 

In yet another embodiment, target-specific reference information associated wrth a 
particular reference target may be transfer^ to an image metrology processor in a more 
automated fashion. For cample, in one embodiment, an automated coding scheme is used to 
transfer target-specific reference information to an image metrology processor. Accordmg to 
one aspect of .his embodiment, a. leas, one automatically readable coded pattern may be 
coupled to the reference target, wherein the automatically readable coded pattern includes 
coded information relating to at leas, one physical property of the reference targe, (e.g., 

dimensions of the reference target and/or one or more ODRs, baseline characteristics of 
detectable properties of the DDKs, sensitivities of the ODRs to rotation, etc.) 

~f*u<* reference tareet 120A shown in Fig. 8. 
Fig 10A illustrates a rear view of the reterence target 

According to one embodiment for transferring target-specific reference information to an 
image metrology processor in a more automated manner, Fig. 10A shows ma, a bar code 129 
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conning coded information may be affixed to a rear surface 127 of the reference target 
12 0A Tire coded information contained in the bar code 129 may include, for example, the 
..get-specific reference information itself, or a serial number tha, uniquely identifies the 
reference target 120A. The serial number in turn may be cross-referenced to .arget-spec.fc 
reference information which is previously stored, for example, in memory or on a storage 
medium of the image metrology processor. 

,„ one aspect of the embodiment show in Fig. 10A, the bar code 129 may be scanned, 
for example, using a bar code reader coupled to the image metrology processor, so as to 
extract and dowrioad the coded information contained in the bar code. Alternatively, m 
another aspect, an image may be obtained of the «ar surface 127 of the target including the 
bar code 129 (e.g„ using me camera 22 shown in Fig. 6), and the image may be analyzed by 
the image metrology processor to extract the coded information. Again, once the ,mage 
metrology processor has access to the target-specific reference information associated wrth a 
particular reference targe, mat targe, may be used repeated* in afferent scenes for whtch 
one or more images are downloaded to the processor for a vanous image metrology purposes. 

• * OonrimA arcordine to one embodiment of the 

With reference again to Figs. 8 and 1UA, according 

invention, the reference target 120A may be fabricated such tha, the ODRs 122A and 122B 
and me fiducial marks 124A-124D are formed as artwork masks ma, are coupled ,o one or 
bom of the fron, surface 121 and me rear surface 127 of an essentially planar substrate 133 
which serves as the body of the reference target. For example, in one aspect of mis 
embodiment, conventional techniques for printing on a solid body may be employed to pnn, 
one or more artwork masks of various reference objects on the substrate 133. According to 
various aspects of mis embodiment, one or more masks may be monolimically formed and 
include a number of reference objects; alternatively, a number of masks including a smglc 
reference object or particular sub-groups of reference objects may be coupled to (e.g., prm.ed 
on) the substrate 133 and arranged in a particular manner. 

Furthermore, in one aspect of mis embodiment, the substrate 133 is essentially 
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.nadeftomoneofavane^ofplas.iCglass.orglass-likematenMs). 
transput (e.g., made from fa ^ ^ 

parucular, Fig. 10A shows tne p. xamIJ .es of reflectors suitable 

■ „f,h« substrate 133 beneath the reflector 131. Examples 
cut-away vrew of the substrate reflective films such as 3M 

for purposes of the invention include, bu, are no, — » . -» - 
. . hV,e™ reflector films, and Lambertian reflectors, such as whrte paper (e.g., 

, laspect^ereflec.ornueflectsradia.ion.atisincidenttothefron, 
printer paper). In thts aspect, ^ 

— ^;t:illr,eiJroneorbothofmeO DRS 

— — SU T a ; e'cl^. — re fl ector 131 coup.edto.e 
172A and 122B may function as "reflective ODKS (i- 

tee .27 of the reference target). Alternative,,, mother embodiment, 
rear surface 127 of the r the 0 DRs 122A and 122B may funcuon 

t^get tha, do not include one or more reflectors 1 3 1 , the ODRs 

^'"rZ:-:— e ntsof.— .areferencetargetmaybe 
d ej t L: i easUnpa rt on*epa rti cu,area m erac,bra«on ta — 

rlonpletersmatanimagemetrologymethodorappara^softhemventron 

tion^whichintummayrelatetomeasurementaccuracy.as 

iCo^::!— edinlgiven.ference.getmaybee.press.intermsof 
application by the relationship 

(18) 

2F >: U-#ODR, 
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WDR is the number of out-of-plane rotations (i.e., pitch and/or yaw) of the reference target 
,ha. may be determined from differently-oriented (e.g., orthogonal) ODRs included in the 
reference target (i.e., *ODR - zero, one, or two), and F is me number of fiducial marks 
included in the reference target. 

The relationship given by Eq. (18) may be understood as follow, Each fiducial mark 
F generates two collinearity equations represented by the expression of Eq. (10), as discussed 
above Typically, each collinearity equation includes at least three unknown position 
parameters and three unknown orientation parameters of the camera exterior orientation (i.e., 
U > 6 in Eq. (17) ), to be determined from a system of collinearity equations in a resection 
process. In this case, as seen from Eq. (18), if no ODRs are included in the reference targe, 
(i e #ODR - 0), at least three fiducial marks Fare required to generate a system of at least 
six collinearity equations in a. leas, six unknowns. This situation is similar to that discussed 
above in connection with a conventional resection process using at least three control points. 

Alternatively, in embodiments of reference targets according to the invention that 
include one or more differently-oriented ODRs, each ODR directly provides orientation (i.e., 
camera bearing) information in an image ma, is related to one of two orientation parameters of 
the camera exterior orientation (i.e. pitch or yaw), as discussed above and in greater detatl m 
Section L of the Detailed Description. Stated differently, by employing one or more ODRs m 
Are reference target, one or two (i.e., pitch and/or yaw) of the three unknown orientation 
parameters of the camera exterior orientation need no, be determined by solving the system of 
collinearity equations in a resection process; rather, these orientation parameters may be 
substituted into the collinearity equations as a previously determined parameter that is denved 
from camera bearing information directly provided by one or more ODRs in an image. In tins 
manner, the number of unknown orientation parameters of the camera exterior orientation to 
be determined by resection effectively is reduced by the number of ou,-of-plane rotations of 
the reference target that may be determined from differently-oriented ODRs included in the 
. reference target. Accordingly, in Eq. (18), the quantity *ODR is subtracted from the number 
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of initially unknown camera calibration parameters U. 

In view of the foregoing, with reference to Eq. (18), the particular example of the 
reference target 120A shown in Fig. 8 (for which F= 4 and #ODR = 2) provides information 
sufficient to determine ten initially unknown camera calibration parameters U. Of course, it 
should be appreciated that if fewer than ten camera calibration parameters are unknown, all of 
the reference objects included in the reference target 120A need not be considered in the 
determination of the camera calibration information, as long as the inequality of Eq. (18) is 
minimally satisfied (i.e., both sides of Eq. (18) are equal). Alternatively, any "excessive" 
information provided by the reference target 120A (i.e., the left side of Eq. (18) is greater than 
the right side) may nonetheless be used to obtain more accurate results for the unknown 
parameters to be determined, as discussed in greater detail in Section L of the Detailed 
Description. 

Again with reference to Eq. (18), other examples of reference targets according to 
various embodiments of the invention that are suitable for determining at least the six camera 
exterior orientation parameters include, but are not limited to, reference targets having three or 
more fiducial marks and no ODRs, reference targets having three or more fiducial marks and 
one ODR, and reference targets having two or more fiducial marks and two ODRs (i.e., a 
generalization of the reference target 120A of Fig. 8). From each of the foregoing 
combinations of reference objects included in a given reference target, it should be 
appreciated that a wide variety of reference target configurations, as well as configurations of 
individual reference objects located in a single plane or throughout three dimensions of a 
scene of interest, used alone or in combination with one or more reference targets, are suitable 
for purposes of the invention to determine various camera calibration information. 

With respect to camera calibration by resection, it is particularly noteworthy that for a 
closed-form solution to a system of equations based on Eq. (10) in which all of the camera 
model and exterior orientation parameters are unknown (e.g., up to 13 or more unknown 
. parameters), control points may not all lie in a same plane in the scene (as discussed in 
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Section F in the Description of the Related Art), .n particular, to so.ve for extensive camera 
caption information (including several or al, of the exterior orientation, interior orientation, 
a„ d iens distortion parameters), some "depth" information is recurred related to a distance 
between the camera (i.e., the camera origin, and the reference target, which informal 
generaliy would not he provided by a number of control points all lying in a same p,ane (e.g., 

on a planar reference target) in the scene. 

,n view of the foregoing, according to another embodiment of the invention, a 
reference targe, is particularly designed to include combinations and arrangements of RFIDs 
and ODRs that enable a determination of extensive camera calibration information usmg a 
single planar reference target in a sing.e image. In particular, according ,0 one aspect of tins 
embodiment, one or more ODRs of the reference target provide information in the image of 
the scene in which the target is placed that is related to a distance between the camera and the 
ODR (and hence the reference target). 

Fig 10B is a diagram illustrating an example of a reference targe, 400 according to 
one embodiment of fte invention that may be placed in a scene to facilitate a determination of 
extensive camera calibration information from an image of the scene. According to one 
aspect of mis embodiment, dimensions of the reference target 400 may be chosen based on a 
particular image metrology application such mat the reference target 400 occupies on the 
order of approximate* 250 pixels by 250 pixels in an image of a scene. It should be 
appreciated, however, mat the particular arrangement of reference objects shown in Fig. 10B 
and tire relative sizes of the reference objects and the target are for purposes of illustration 
only and that the invention is not limited in these respects. 

' The reference target 400 of Fig. 10B includes four fiducial marks 402A-402D and two 
ODRs 404A and 404B. Fiducial marks similar to those shown in Fig. 10B arc discussed m 
detail in Sections G3 and K of the Detailed Description. In particular, according to one 
embodiment, the exemplary fiducial marks 402A-402D shown in Fig. 10B facilitate automatic 
detection of the reference target 400 in an image of a scene containing the target. The ODRs 
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404A and 404B shown in Fig. 10B are discussed in detail in Sections G2 and J of the Detailed 
Description. In particular, near-field effects of the ODRs 404A and 404B that facilitate a 
determination of a distance between the reference target 400 and a camera obtaining an image 
of the reference target 400 are discussed in Sections G2 and J of the Detailed Description. 
Exemplary image metrology methods for processing images containing the reference target 
400 (as well as the reference target 120A and similar targets according to other embodiments 
of the invention) to determine various camera calibration information are discussed in detail in 
Sections H and L of the Detailed Description. 

Fig. 10C is a diagram illustrating yet another example of a reference target 1020A 
according to one embodiment of the invention. In one aspect, the reference target 1020A 
facilitates a differential measurement of orientation dependent radiation emanating from the 
target to provide for accurate measurements of the target rotations 134 and 136. In yet 
another aspect, differential near-field measurements of the orientation dependent radiation 
emanating from the target provide for accurate measurements of the distance between the 

target and the camera. 

Fig. 10C shows that, similar to the reference target 120A of Fig. 8, the target 1020A 
has a geometric center 140 and may include four fiducial marks 124A-124D. However, 
unlike the target 120A shown in Fig. 8, the target 1020A includes four ODRs 1022A-1022D, 
which may be constructed similarly to the ODRs 122A and 122B of the target 120A (which 
are discussed in greater detail in Sections G2 and J of the Detailed Description). In the 
embodiment of Fig. 10C, a first pair of ODRs includes the ODRs 1022A and 1022B, which 
are parallel to each other and each disposed essentially parallel to the *, axis 138. A second 
pair of ODRs includes the ODRs 1022C and 1022D, which are parallel to each other and each 
disposed essentially parallel to the y, axis 132. Hence, in this embodiment, each of the ODRs 
1022A and 1022B of the first pair emanates orientation dependent radiation that facilitates a 
determination of the yaw rotation 136, while each of the ODRs 1022C and 1022D of the 
second pair emanates orientation dependent radiation that facilitates a determination of the 
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pitch angle 134. acWO DRofthe orthogonal pairs ofODRs shown in 

According to one em—,, ° rf ^ pair ^ at lea st one detectab.e 

,,.00, — - — Ihlproper.o.rheolherOOK 

~ , about its secondary axis, 

^^ing^rs^ample^awording to one embodiment, ^ ' n ^^^^^'^ t | i ^ 1 | > j3^ e 

causes a position of a radiation spot 1026 A o ^ ^ ^ rf ^ 

^^"^ ; ^ mOT upwardalong^e longitudinal axisof 
Eradiation spot 1026C of the ODR 1022C P ^ ^ 

*» ODR 1022D to move downwar ^ ^ ^ ^ (e g „ as 

rotations of the reference target 1020A Can ^ ^ ^ yaw 

shoul dbeappreeiated,however,that,hts ffl4 « other 

detectablepropertiesofanODRCe-g-, P a more detailed 

. example of an ODR pair in «hrch each ODR ts eons 
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of differential effects is discussed below in Sections G2 and J of the Detailed 



G 2. Exemplary Orientation-Dependent Radiation Sources (ODRs) 
As discussed above, according to one embodiment of the invention, an orientation- 
dependent radiation source (ODR) may serve as a reference object in a scene of interest (e.g., 
as exemplified by the ODRs 122A and 122B in the reference target 120A shown in Fig. 8). In 
general, an ODR emanates radiation having at least one detectable property (which is capable 
of being detected from an image of the ODR) that varies as a function of a rotation (or 
alternatively "viewing angle") of the ODR. In one embodiment, an ODR also may emanate 
radiation having at least one detectable property that varies as a function of an observation 
distance from the ODR (e.g., a distance between the ODR and a camera obtaining an image of 
the ODR). 

A particular example of an ODR according to one embodiment of the invention is 
discussed below with reference to the ODR 122A shown in Fig. 8. It should be appreciated, 
however, that the following discussion of concepts related to an ODR may apply similarly, for 
example, to the ODR 122B shown in Fig. 8, as well as to ODRs generally employed in 
various embodiments of the present invention. 

As discussed above, the ODR 122A shown in Fig. 8 emanates orientation-dependent 
radiation 126A from an observation surface 128A. According to one embodiment, the 
observation surface 128A is essentially parallel with the front surface 121 of the reference 
target 120A. Additionally, according to one embodiment, the ODR 122A is constructed and 
arranged such that the orientation-dependent radiation 126A has at least one detectable 
property that varies as a function of a rotation of the ODR 122A about the secondary axis 132 

passing through the ODR 122 A. 

According to one aspect of this embodiment, the detectable property of the orientation- 
dependent radiation 126 A that varies with rotation includes a position of the spatial 
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ODR 122A. For example, Fig. 8 shows «ha., according to this aspect, as the ODR 122A ,s 
rotated about .he secondary axis 132, the position of the spatiai distribution of the radiation 
]2 6A moves from left to right or vice versa, depending on the direction of rotation, tn a 
direction parallel to the primary axis ,30 (as indicated by tire opposite* directed arrows 
shown schematically on the observation surface 128A). According to various other aspects of 
this embodiment, a spatiai period ofthe orientation-dependent radiation ,26A (e.g., a distance 
between adjacent oval-shaped radiation spots shown in Fig. 8), a portion of the 
orientation-dependent radiation 126A, a„d/or a wavelengtir ofthe orientation-dependent 
radiation 126A, may vary with rotation of the ODR 122A about the secondary 

Figs 11 A, 1 IB, and 1 1C show various views of a particular example ofthe ODR 
ntK suitab.e for use in the reference target 120A shown in Fig. 8, according to one 
embodiment of the invention. As discussed above, an ODR similar to tha, shown in Fig, 

, ^ r»r»T> i ->or r,f the reference target 120 A shown in Fig. 8, as 
1 1 A-C also may be used as the ODR 122B ot the reierence g 

wei, as in various omer embodiments ofthe invention, .n one aspect, the ODR 122A shown 
in Figs 1 1 A-C may be constructed and arranged as described in U.S. Paten, No. 5,936,723, 
entitled "Orientation Dependent Reflector, - hereby incorporated herein by reference, or may 
be constructed and arranged in a manner similar to tha, described in this reference. In other 
aspects tire ODR 122A may be constructed and arranged as described m U.S. Patent 
Application Serial No. 09/3 .7,052, Med May 24, 1999, entitled "Orientation-Dependen, 
Radiation Source," also hereby incorporated herein by reference, or may be constructed and 
arranged in a manner similar to that described inthis reference. A detailed mathematica. and 
geometric analysis and discussion of ODRs similar to that shown in Fig, 1 1 A-C is presented 
in Section J ofthe Detailed Description. 

Fig llAisa front view ofthe ODR 122A, looking on to the observation surface USA 
at a normal viewing angle (i.e., perpendicular ,o tire observation surface), in which the 
primary axis 130 is indicated horizontally. Fig. 1 IB is an enlarged front view of a portion of 
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purposes of this disCosure, a norma, viewing ang,e of*. ODR *-* <-* * 

considered as a 0 degree rotation. 

Figs U A-H C show that, according to one embodiment, the ODR 122A .eludes a 
fir st g ratingl4 2 andasecond g ratin g 144. Each of the first and second gratings indude 
s.hstantiaUyopa.ueregionsseparatedhysu— — 

. pa Hf the first grating 142 includes substantially opaque regions 226 
with reference to Fig. 1 1 C, the nrsx gr<umg 

« -.r t.HinFiss 1 1A-1 1C as areas filled with dots) which are separated by 
f generally indicated in Mgs. ha 

+ -^998 Similarly the second grating 144 includes 
openings or substantially transparent regions 228. Similarly, 

• ™ 999 Generally indicated in Figs. 1 1 A-l 1C by areas shaded with 
substantially opaque regions 222 (generally ina 

veriica, lines, which are separated by openings or subs.an.ia,., transparent reg,ons 230 The 
opaque regions of eaeh gra.in g may be made of a variety of materia, that a. .east partly 
absorb, or do not fuUy — a particular waveleng* range or ranges of radiafon. It 
should be appreciated that the par.icu.ar restive arrangement and spacing of respective 
opaque and transparent regions for the gratings 142 and ,44 shown in Figs. . , A-, 1C ,s fcr 
purposes of illusion only, and tha, a number of arrangement and spacing* are poss.ble 
according to various embodiments of the invention. , „ nnp , A 

In 0 „e embodiment, the f,rs, g^ng 142 and the second grating 144 of the ODR 12 A 

hOT ing athiCoress ,47. ,n one aspect of this embodiment, the ODR !22A may be fabr.ca.ed 
using convention* semiconductor fabrication techniques, in which the firs, and second 
stings are each formed by patterned thin films (e.g., of materia, tha, a, leas, pariiaUy absorbs 
nation a, one or more appropriate wavelengths) disposed on opposite sides of the 
substantially transparent substrate ,46. In another aspect, convention, techniques for 
printing on a solid body may be employed to print the first and second gratings on .he 
substrate 146. ,n par.icu.ar, i. should be apprecia,ed that in one embodiment, the substrate 
,46 of the ODR ,22A shown in Figs. 1 , A-, 1C coincides with (i.e., is the same as) the 
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1 33 It should be appreciated, however, that the invention is not limited in this respect, as 
P0SSit l canoe seen mF i B ,llA^ 

of rhe ODR 122 A essentially defines the observation surface ,2SA. Accordingly, in uus 

« ^ ^ fc «^^>.^--• W •^'* ,b,, - , - ^,1 " ,,,,, 

.naybereferredtoasa-bacrgratingoftheODR. Additionally, according to one 

frequencies (e.g., in cycles/metcr); name.y either one or both of the subs— opaque 
Jonsandthesubstanti ally parent region, of one grating nray have different drrnensrons 
ton the corresponding regions of the other grain, As a result of the different spatial 

• rfrhe thickness 147 of the transparent substrate 146, the radiauon 
frequencies ofthegraungs and the thickness 14/0 v 

emission properties of the ODR 122A depends on a particular rotation 136 of the ODR 

^L-^^^^^•^«■* rf,, ■ oDB,,,-, "•' 

normal to the observation surface 128 A). 

For example, with reference to Fig. 11 A, at a zero degree rotation (i.e., a norma, 

fig u re ,ra d ia,i„nessen,ial,yisbloc k edinacen te rpor,ionof,eODR122A, W hereasthOD R 

becomes squally more transmissive moving away from the center portion, as mdrcated ,n 
Fig UAbyclearregionsbetweenthegratings. As the ODR 122A is rotated about the axis 

12 SAchange. This phenomenon may be explained withthe assistance of Figs. , Aan 2B, 
and is discussed in detail in Section J of the Detailed Description. Bom Figs. 12A and 12B 
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are top views of a portion of the ODR 122A, similar to that shown in Fig. 1 1C. 

In Fig. 12A, a central region 150 of the ODR 122A (e.g., at or near the reference point 
125A on the observation surface 128A) is viewed from five different viewing angles with 
respect to a normal to the observation surface 128A, represented by the five positions A, B, C, 
D, and E (corresponding respectively to five different rotations 136 of the ODR about the axis 
132, which passes through the central region 150 orthogonal to the plane of the figure). From 
the positions A and B in Fig. 12A, a "dark" region (i.e., an absence of radiation) on the 
observation surface 128 A in the vicinity of the central region 150 is observed. In particular, a 
ray passing through the central region 150 from the point A intersects an opaque region on 
both the first grating 142 and the second grating 144. Similarly, a ray passing through the 
central region 150 from the point B intersects a transparent region of the first grating 142, but 
intersects an opaque region of the second grating 144. Accordingly, at both of the viewing 
positions A and B, radiation is blocked by the ODR 122A. 

In contrast, from positions C and D in Fig. 12A, a "bright" region (i.e., a presence of 
radiation) on the observation surface 128A in the vicinity of the central region 150 is 
observed. In particular, both of the rays from the respective viewing positions C and D pass 
through the central region 150 without intersecting an opaque region of either of the gratings 
142 and 144. From position E, however, a relatively less "bright" region is observed on the 
observation surface 128A in the vicinity of the central region 150; more specifically, a ray 
from the position E through the central region 150 passes through a transparent region of the 
first grating 142, but closely intersects an opaque region of the second grating 144, thereby 
partially obscuring some radiation. 

Fig. 12B is a diagram similar to Fig. 12A showing several parallel rays of radiation, 
which corresponds to observing the ODR 122A from a distance (i.e., a far-field observation) 
at a particular viewing angle (i.e., rotation). In particular, the points AA, BB, CC, DD, and 
EE on the observation surface 128A correspond to points of intersection of the respective far- 
field parallel rays at a particular viewing angle of the observation surface 128A. From Fig. 
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12B it can be seen that the surface points AA and CC would appear "brightly illuminated 
0 e a more intense radiation presence) a. this viewing angle in the far-field, as the respective 
parallel rays passing through these points intersect transparent regions of both the firs, grafng 
,42 and .he second grating 144. In contrast, the points BB and EE on the observation surface 
128A would appear "dark" (i.e., no radiation) at this viewing angle, as the rays passing 
through these points respectively intersect an opaque region of the second grading 144. The 
point DD on the observation surface 128A may appear "dimly" illuminated a, mis viewing 
angle as observed in the far-field, because the ray passing through the point DD nearly 
intersects an opaque region of the second grating 144. 

Thus, from the foregoing discussion in connection with both Figs. 12A and 12B, it 
may be appreciated that each point on the observation surface 128A of the orientation- 
dependent radiation source 122A may appear ••brightly" illuminated from some viewing 
angles and "dark" from other viewing angles. 

According to one embodiment, the opaque regions of each of the first and second 
gratings 142 and 144 have an essentially rectangular shape. In this embodiment, the spatial 
distribution of the orientation-dependent radiation 126A observed on the observation surface 
128A of the ODR 122A may be understood as the product of two square waves. In particular, 
the relative arrangement and different spatia! frequencies of the first and second gratings 
produce a "Moire" pattern on the observation surface 128A that moves across the observation 
surface 128AastheODR 122A is rotated about the secondary axis 132. A Moire pattern 
type of interference pattern mat occurs when two similar repeating patterns are almost, bu, no, 
quite, me same frequency, as is me case with the first and second gratings of the ODR 122A 
according to one embodiment of the invention. 

Figs. 13A, 13B, 13C, and 13D show various graphs of transmission characteristics of 
the ODR 122A at a particular rotation (e.g., zero degrees, or normal viewing.) In Figs. 13A- 
1 3D a relative radiation transmission level is indicated on the vertical axis of each graph, 
while a distance (in meters) along the primary axis ,30 of the ODR 122A is represented by 
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,h= horizontal axis of each graph. In particular, the ODR reference point 125A is indicated a, 
x - 0 along the horizontal axis of each graph. 

The graph of Fig. 13A shows two plots of radiation transmission, each plot 
corresponding to the transmission through one of the two gratings of the ODR 122A if the 
grating were used alone. In particular, the legend of the graph in Fig. 13A indicates that 
radiation transmission tough a "front" grating is represented by a solid line (which in this 
example corresponds to the first grating 142) and through a "back" grating by a dashed line 
(which in tins example corresponds to the second grating 144). In the example of Fig. 13A, 
the first grating 142 (i.e., the front grating) has a spatial frequency of 500 cycles per meter, 
and the second grating 144 (i.e., the back grating) has a spatial frequency of 525 cycles per 
meter. It should be appreciated, however, tha, the invention is no, limited in this respect, and 
mat these respective spatial frequencies of the gratings are used here for purposes of 
iUustration only. In particular, various relationships between the front and back grating 
frequency may be exploited to achieve near-field and/or differential effects from ODRs, as 
discussed further below in this section and in Section J of the Detailed Description. 

The graph of Fig. 13B represents the combined effect of the two gratings at the 
particular rotation shown in Fig. 13A. In particular, me graph of Fig. I3B shows a plot 126A' 
of the combined transmission characteristics of the first and second gratings along the pnmary 
axis 130 of the ODR over a distance of ±0.01 meters from the ODR reference point 125A. 
The plot 126A' may be considered essentially as the product of two square waves, where each 
square wave represents one of the first and second gratings of the ODR. 

The graph of Fig. 13C shows the plot 126A' using a broader horizontal scale than the 
graphs of Figs. 13A and 13B. In particular, whereas the graphs of Figs. 13A and 13B 
Ulustrate radiation transmission characteristics over a lateral distance along the primary axis 
130 of ±0.01 meters from the ODR reference point 125 A, the graph of Fig. 13C illustrates 
radiation transmission characteristics over a lateral distance of ±0.05 meters from the 
reference point 125A. Using the broader horizontal scale of Fig. 13C, it is easier to observe 
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the Moire pattern that is generated due ,0 the different spatial frequencies of the firs, (front) 
and second (back) gratings of the ODR 122A (shown in the graph of Fig. 13A). The Moire 
pattern shown in Fig. 13C is somewhat related to a pulse-width modulated signal, bu, drffers 
from such a signal in that neither the boundaries nor the centers of the individual rectangular 
"pulses" making up the Moire pattern are perfectly periodic. 

I„ the graph of Fig. 13D, the Moire pattern shown in the graph of Fig. 13C has been 
tow-pass filtered (e.g., by convolution with a Gaussian having a -3 dB frequency of 
approximately 200 cycles/meter, as discussed in Section , of the Detailed Description) to 
iUustrate the spatial distribution (i.e., essentially a triangular waveform) of orientatton- 
dependen. radiation 126A that is ultimately observed on the observation surface 128A of the 
ODR 122A From the filtered Moire pattern, the higher concentrations of radiafon on the 
observation surface appear as three peaks 1S2A. 152B. and 152C inthe graph of Fig. 13D, 
which may be symbolically represented by three "centroids" of radiation detectable on the 
observation surface 128A (as illustrated for example in Fig. 8 by the three oval-shaped 
radiation spots). As shown in Fig. BD.aperiod 15 4ofmetriangularwaveformrepresen,,ng 
the radiation .26A is approximately 0.04 meters, corresponding to a spatial frequency of 
approximately 25 cycles/meter (i.e., the difference between me respective front and back 

grating spatial frequencies). 

As may be observed from Figs. 13A-13D, one interesting attribute of the ODR 122A 
is that a transmission peak in the observed radiation ! 26A may occur a, a location on the 
observation surface 128A tha, corresponds to an opaque region of one or bom of the gratmgs 
,42 and 144. For example, with reference to Fig, 13B and 13C, the unfiltered Moire pattern 
U6A- indicates zero transmission a, , - 0; however, the filtered Moire pattern 126A shown » 
Fig 13D tadica«esatran S missionpeakl52Bat* = 0. This phenomenon is primanly a 
consequence of filtering; in particular, the high frequency components of the signa! !26A' 
corresponding to each of the gratings are nearly removed from the signal 126A, leavmg 
behind an overall radiation density corresponding to a cumulative effect of radratron 
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transmitted through a number of gratings. Even in the filtered signal 126A, however, some 
artifacts of the high frequency components may be observed (e.g., the small troughs or ripples 
along the triangular waveform in Fig. 1 3D.) 

Additionally, it should be appreciated that the filtering characteristics (i.e., resolution) 
of the observation device employed to view the ODR 122A may determine what type of 
radiation signal is actually observed by the device. For example, a well-focussed or high 
resolution camera may be able to distinguish and record a radiation pattern having features 
closer to those illustrated in Fig. 13C. In this case, the recorded image may be filtered as 
discussed above to obtain the signal 126A shown in Fig. 13D. In contrast, a somewhat 
defocused or low resolution camera (or a human eye) may observe an image of the orientation 
dependent radiation closer to that shown in Fig. 13D without any filtering. 

With reference again to Figs. 1 1 A, 12A, and 12B, as the ODR 122A is rotated about 
the secondary axis 132, the positions of the first and second gratings shift with respect to one 
another from the point of view of an observer. As a result, the respective positions of the 
peaks 152A-152C of the observed orientation-dependent radiation 126A shown in Fig. 13D 
move either to the left or to the right along the primary axis 130 as the ODR is rotated. 
Accordingly, in one embodiment, an orientation (i.e., a particular rotation angle about the 
secondary axis 132) of the ODR 122 A is related to the respective positions along the 
observation surface 128A of one or more radiation peaks 152A-152C of the filtered Moire 
pattern. If particular positions of the radiation peaks 152A-152C are known a priori with 
respect to the ODR reference point 125A at a particular "reference" rotation or viewing angle 
(e.g., zero degrees, or normal viewing), then arbitrary rotations of the ODR may be 
determined by observing position shifts of the peaks relative to the positions of the peaks at 
the reference viewing angle (or, alternatively, by observing a phase shift of the triangular 
waveform at the reference point 125 A with rotation of the ODR). 

With reference to Figs. 1 1 A, 1 1C, 12A and 12B, it should be appreciated that a 
horizontal length of the ODR 122A along the axis 130, as well as the relative spatial 
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frequencies of the first grating 142 and the second grating 144, may be chosen such that 
different numbers of peaks (other than three) in fte spatiai distribution of the orientation- 
dependent radiation 126A shown in Fig. 13D may be visible on the observation surface a, 
various rotations of the ODR. In particular, the ODR 122A may be constructed and arranged 
such that only one radiation peak is detectable on the observation surface 128A of ft. source 
at any given rotation, or several peaks are detectable. 

Additionally, according to one embodiment, the spatial frequencies of the firs, grating 
142 and the second grating 144, each may be particularly chosen * resul, in a particular 
direction along the primary axis of ft. ODR for the change in position of the spatial 
distribution of the orientation-dependent radiation with rotation about the secondary axts. For 
example, a back grating frequency higher than a front grating frequency may dictate a first 
direction for the change in position with rotation, while a back grating frequency lower than a 
front grating frequency may dictate a second direction opposite to the firs, direction for the 
change in position with rotation. This effect may be exploited using a pair of ODRs 
constructed and arranged to have opposite directions for a change in position with the same 
rotation to facilitate differentia, measurements, as discussed above in Section Gl of the 
Detailed Description in connection with Fig. IOC. 

Accordingly, it should be appreciated that the foregoing discussion of ODRs is for 
purposes of illusion only, and that the invention is not limited to the particular manner of 
implementing and utilizing ODRs as discussed above. Various effects resulting from 
particular choices of grating frequencies artd other physical characteristics of an ODR are 
discussed further below in Section J of the Detailed Description. 

According to another embodiment, an ODR may be constructed and arranged so as to 
emanate radiation having a, leas, one detectable property that facUitates a determination of an 
observation distance a, which the ODR is observed (e.g., the distance between the ODR 
reference point and the origin of a camera which obtains an image of the ODR). For example, 
according to one aspect of this embodiment, an ODR employed in a reference target simtlar to 
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the reference target 120A shown in Fig. 9 may be constructed and arranged so as to facilitate a 
determination of the length of the camera bearing vector 78. More specifically, according to 
one embodiment, with reference to the ODR 122A illustrated in Figs. 1 1 A-l 1C, 12A, 12B and 
the radiation transmission characteristics shown in Fig. 13D, a period 154 of the orientation- 
dependent radiation 126A varies as a function of the distance from the observation surface 
128A of the ODR at a particular rotation at which the ODR is observed. 

In this embodiment, the near-field effects of the ODR 122A are exploited to obtain 
observation distance information related to the ODR. In particular, while far-field observation 
was discussed above in connection with Fig. 12B as observing the ODR from a distance at 
which radiation emanating from the ODR may be schematically represented as essentially 
parallel rays, near-field observation geometry instead refers to observing the ODR from a 
distance at which radiation emanating from the ODR is more appropriately represented by 
non-parallel rays converging at the observation point (e.g., the camera origin, or nodal point 
of the camera lens system). One effect of near-field observation geometry is to change the 
apparent frequency of the back grating of the ODR, based on the rotation of the ODR and the 
distance from which the ODR is observed. Accordingly, a change in the apparent frequency 
of the back grating is observed as a change in the period 154 of the radiation 126A. If the 
rotation of the ODR is known (e.g., based on far-field effects, as discussed above), the 
observation distance may be determined from the change in the period 1 54. 

Both the far-field and near-field effects of the ODR 122A, as well as both far-field and 
near-field differential effects from a pair of ODRs, are analyzed in detail in Section J of the 
Detailed Description and the figures associated therewith. An exemplary reference target 
particularly designed to exploit the near-field effects of the ODR 122A is discussed above in 
Section Gl of the Detailed Description, in connection with Fig. 10B. An exemplary reference 
target particularly designed to exploit differential effects from pairs of ODRs is discussed 
above in Section Gl of the Detailed Description, in connection with Fig. IOC. Exemplary 
. detection methods for detecting both far-field and near-field characteristics of one or more 
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Description and the figures associated therewith. 

G3. Exemplary Fiducial Marks and exemplary methods for 
detecting such marks 

As discussed above, one or more fiducial marks * **** >" * — ""^ 
as reference objects for which reference —ion is known a P Hor, For example, » 

M^mar.saresui.ab.eforpurposesof.he — according to vartous embodmrents, as 
discussed further below. 

Z^thatfa la.es either nranua, or automatic Uonof •henrarUnan.ma.e 

no, limited ,o a shape of «he mark (e.g., a particular polygon form or penmeter shape), a 

pattern.oranycombinauonorsubse.offteforegoingpropert.e, 

^ metrol ogy applications), and methods for detecting such marks. For purposes of th s 
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stated differently, a robust mark has one or more unique detectable properties in an image that 
do not change as a function of the size of the mark as it appears in the image, and/or an 
orientation (rotation) and position (translation) of the mark with respect to a camera (i.e., a 
viewing angle of the mark) as an image of a scene containing the mark is obtained. In other 
aspects, a robust mark preferably has one or more invariant characteristics that are relatively 
simple to detect in an image, that are unlikely to occur by chance in a given scene, and that are 
relatively unaffected by different types of general image content. These properties generally 
facilitate automatic identification of the mark under a wide variety of imaging conditions. 

In a relatively straightforward exemplary scenario of automatic detection of a mark in 
an image using conventional machine vision techniques, the position and orientation of the 
mark relative to the camera obtaining the image may be at least approximately, if not more 
precisely, known. Hence, in this scenario, the shape that the mark ultimately takes in the 
image (e.g., the outline of the mark in the image) is also known. However, if this position and 
orientation, or viewing angle, of the mark is not known at the time the image is obtained, the 
precise shape of the mark as it appears in the image is also unknown, as this shape typically 
changes with viewing angle (e.g., from a particular observation point, the outline of a circle 
becomes an ellipse as the circle is rotated out-of-plane so that it is viewed obliquely, as 
discussed further below). Generally, with respect to conventional machine vision techniques, 
it should be appreciated that the number of unknown parameters or characteristics associated 
with the mark to be detected (e.g., due to an unknown viewing angle when an image of the 
mark is obtained) significantly impacts the complexity of the technique used to detect the 
mark. 

Conventional machine vision is a well-developed art, and the landmark detection 
problem has several known and practiced conventional solutions. For example, conventional 
"statistical" algorithms are based on a set of characteristics (e.g. area, perimeter, first and 
second moments, eccentricity, pixel density, etc.) that are measured for regions in an image. 
The measured characteristics of various regions in the image are compared to predetermined 
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vah.es for these characteristics tha, identify the presence of a mark, and close matches are 
sough. Alternative*, in convention. "temp>a«e matching" algorithms, a template for a mark 
is stored on a storage medium (e.g., in tire memory of the processor 36 shown in Fig. 6), and 
various regions of an image are searched ,0 seek matches to the stored template. Typrcally, 
the computational costs for such a lg ori,hms are quite high. In particular, a number of 
different templates may need to be stored for comparison with each region of an unage to 
account for possibly different viewing angles of tire mark relative to tire camera (and hence a 
number of potentially different shapes for the mark as it appears in tire image). 

Yet other examples of conventional machine vision algorithms employ a Hough 
Transform, which essentially.describes a mapping from image-space to shape-space. In 
algorithms employing the Hough Transform, the "dimensionality" of the shape-space is gtven 
by tire number of parameters needed to describe all possible shapes of a mark as it mtght 
appear in an image (e.g., accounting for a variety of different possible viewing ang,es of the 
nrark with respect to tire camera). Generally, the Hough Transform approach is somewhat 
computationally less expensive than template matching algorithms. 

The foregoing examples of conventional machine vision detection algorithms 
general* may be classified based on whether they operate on a very small region of an image 
(.•point" algorithms), involve a scan of a portion of tire image along a line or a curve ropen 
curve" algorithms), or evaluate a larger area region of an image ("area" algorithms). In 
general, the more pixels of a digital image that are evaluated by a given detection algorithm, 
the more robust tire results are with respect to noise (background content) in the image; m 
particular, algorithms tha, operate on a greater number of pixels generaUy are more efficient a, 
rejecting false positives (i.e., incorrect identifications of a mark). 

For example, "point" algorithms generally involve edge operators tirat detect vanous 
properties of a point in an image. Due to tire discrete pixe! nature of digital images, pom, 
algorithms typically operate on a small region comprising 9 pixels (e.g., a 3 pixel by 3 p.xe. 
area) In these algorithms, the Hough Transform is often applied to pixels detected wtth an 



-95- 

edge operator. Alternatively, in "open curve" algorithms, a one-dimensional region of the 
image is scanned along a line or a curve having two endpoints. In these algorithms, generally 
a greater number of pixels are grouped for evaluation, and hence robustness is increased over 
point algorithms (albeit at a computational cost). In one example of an open curve algorithm, 
the Hough Transform may be used to map points along the scanned line or curve into shape 
space. Template matching algorithms and statistical algorithms are examples of "area- 
algorithms, in which image regions of various sizes (e.g., a 30 pixel by 30 pixel region) are , 
evaluated. Generally, area algorithms are more computationally expensive than point or curve 
algorithms. 

Each of the foregoing conventional algorithms suffer to some extent if the scale and 
orientation of the mark that is searched for in an image are not known a priori. For example, 
statistical algorithms degrade because the characteristics of the mark (i.e., parameters 
describing the possible shapes of the mark as it appears in the image) co-vary with viewing 
angle, relative position of the camera and the mark, camera settings, etc. In particular, the 
larger the range that must be allowed for each characteristic of the mark, the greater the 
potential number of false-positives that are detected by the algorithm. Conversely, if the 
allowed range is not large enough to accommodate variations of mark characteristics due, for 
example, to translations and/or rotations of the mark, excessive false-negatives may result. 
Furthermore, as the number of unknown characteristics for a mark increases, template 
matching algorithms and algorithms employing the Hough Transform become intractable (i.e., 
the number of cases that must be tested may increase dramatically as dimensions are added to 
the search). 

Some of the common challenges faced by conventional machine vision techniques 
such as those discussed above may be generally illustrated using a circle as an example of a 
feature to detect in an image via a template matching algorithm. With respect to a circular 
mark, if the distance between the circle and the camera obtaining an image of the circle is 
known, and there are no out-of-plane rotations (e.g., the optical axis of the camera is 
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orth o g o„a, to the plane of the circle), locating the circle in the image requires reso,vm g two 
axis anda,-axi S de E ne S *=p 1 ancof*ec ir d=,.Ifaconve„Uona ltem p 1 a ttm a«o 1 un g 

ira a g e, for example, then .0,000 (i.e., 100>) test conditions are required to determme the , and 

y coordinates of the center of the circle. 

However, if the distance between the circular mark and the camera is unknown, toe 

center of the circle and the radius , of the circle, which chan g es in the image accordm g to the 
ais^ce between the circie arrd the camera. According, a conventional template matchrng 

If each of these dimensions is tested by such an al.orithm a, 100 points, 1 nulhou (,e., ,00 ) 

test conditions are required. 

As discussed above, if a mark is arbitrarily oriented and positioned w lt h respect the 

pla ne of the mark at normal viewin g , such tha, the m*k is viewed obliquely,, the cha,len g e of 
fmding me mark in an ima g e .rows exponent In g eneral, two ou,-of-p.ane — are 

example of me circular mark introduced above, one or more out-of-p.ane rotattons transform 
te circular mark into an ellipse and rotate the major axis of the ellipse to an unknown 

orientation. 

One consequence of such out-of- P la„e rotations, or oblique viewing angles, 
circular mark is to expand the number of dimensions mat a convention* template matchin g 
algorithm (as well as algorithms employing the Hou.h Transform, for example) must search 
t0 five dimensions; namely, * and y coordinates of the center of the circle, a le ng * 
major axis of the elliptical image of the rotated circle, a lensth of the minor axis of the 
elliptical image of the rotated circle, and the rotation of the major axis of the elliptical image 



97 



of the rotated circle. The latter three dimensions or parameters correspond via a complex 
mapping to a pitch rotation and a yaw rotation of the circle, and the distance between the 
camera and the circle. If each of these five dimensions is tested by a conventional template 
matching algorithm at 100 points, 10 billion(i.e., 100 s ) test conditions are required. 
Accordingly, it should be appreciated that with increased dimensionality (i.e., unknown 
parameters or characteristics of the mark), the conventional detection algorithm quickly may 
become intractable; more specifically, in the current example, testing 100 5 templates likely is 
impractical for many applications, particularly from a computational cost standpoint. 

Conventional machine vision algorithms often depend on properties of a feature to be 
detected that are invariant over a set of possible presentations of the feature (e.g., rotation, 
distance, etc). For example, with respect to the circular mark discussed above, the property of 
appearing as an ellipse is an invariant property at least with respect to viewing the circle at an 
oblique viewing angle. However, this property of appearing as an ellipse may be quite 
complex to detect, as illustrated above. 

In view of the foregoing, one aspect of the present invention relates to various robust 
marks that overcome some of the challenges discussed above. In particular, according to one 
embodiment, a robust mark has one or more detectable properties that significantly facilitate 
detection of the mark in an image essentially irrespective of the image contents (i.e., the mark 
is detectable in an image having a wide variety of arbitrary contents), and irrespective of 
position and/or orientation of the mark relative to the camera (i.e., the viewing angle). 
Additionally, according to other aspects, such marks have one or more detectable properties 
that do not change as a function of the size of the mark as it appears in the image and that are 
very unlikely to occur by chance in an image, given the possibility of a variety of imaging 

conditions and contents. 

According to one embodiment of the invention, one or more translation and/or rotation 
invariant topological properties of a robust mark are particularly exploited to facilitate 
detection of the mark in an image. According to another embodiment of the invention, such 
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properties are exploited by employing detection algorithms that detect a presence (or absence) 
of the mark in an image by scanning at least a portion of the image along a scanning path 
(e.g., an open line or curve) that traverses a region of the image having a region area that is 
less than or equal to a mark area (i.e., a spatial extent) of the mark as it appears in the image, 
such that the scanning path falls within the mark area if the scanned region contains the mark. 
In this embodiment, all or a portion of the image may be scanned such that at least one such 
scanning path in a series of successive scans of different regions of the image traverses the 
mark and falls within the spatial extent of the mark as it appears in the image (i.e., the mark 
area). 

According to another embodiment of the invention, one or more translation and/or 
rotation invariant topological properties of a robust mark are exploited by employing 
detection algorithms that detect a presence (or absence) of the mark in an image by scanning 
at least a portion of the image in an essentially closed path. For purposes of this disclosure, an 
essentially closed path refers to a path having a starting point and an ending point that are 
either coincident with one another, or sufficiently proximate to one another such that there is 
an insignificant linear distance between the starting and ending points of the path, relative to 
the distance traversed along the path itself. For example, in one aspect of this embodiment, an 
essentially closed path may have a variety of arcuate or spiral forms (e.g., including an 
arbitrary curve that continuously winds around a fixed point at an increasing or decreasing 
distance). In yet another aspect, an essentially closed path may be an elliptical or circular 
path. 

In yet another aspect of this embodiment, as discussed above in connection with 
methods of the invention employing open line or curve scanning, an essentially closed path is 
chosen so as to traverse a region of the image having a region area that is less than or equal to 
a mark area (i.e., a spatial extent) of the mark as it appears in the image. In this aspect, all or a 
portion of the image may be scanned such that at least one such essentially closed path in a 
series of successive scans of different regions of the image traverses the mark and falls within 
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ll essentially closed path is a circular path, and a radius of a circular pa* is sdK ^^' 

^ u„ ^terted as it appears in the image. 

ri^^cdcecion^n^— 

„ f * image by samplin g a piuramy of pixels disposed in a respective scannm g pa* ca 

bot h open line or curve as we,, as essentially closed pa* scanning .echninues may be 
emp.oyed, alone or in combination, ,o scan an ima g e. Furthermore, some invanan. 

al.ema.ive ,o, open line or curve and/or essentially dosed pa.h scanning me.hods. 

mark isprese,man tag ehav i n g ana r ^ima g econ.en t ,andaUeas.aportino f .he 
Jsepa I a Kl yidentif 1 a W efea.uresof,ema I MhcmarHscapa b ,eofhein g de t ec.ed,an 

viewin g an g .e a. which .he number of separably identify re g ions of ft. mar, can he 
distin g uished (e,., any angle less .han 90 degrees). More specifically, accordin g «o one 
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, 0 anormal,otnemarkofa<least2 5 degrees. In one aspect of this embodiment, the 
separate* identifiable features are disposed with respee, to eaeh other such that the mark , 
capable of being detected at an obiioue viewing angle of a, leas, 30 degree, In yet another 
aspect, the separately identifiable features are disposed win, respect to each other such ti,a. the 
mark is capable of being detected a. an obiioue viewing angle of a. leas. 45 degrees. In ye. 
another aspect, .he separately identifiable features are disposed wifh respec. .o each other such 

One example of « invariant topological property of a mark according to one 
embodiment of the invention includes a particular ordering of various regions or features, or 
an "ordinal property," of the mark. In particular, an ordinal property of a mark refers to a 
unio.ue seo.uen.ia, order of at least three separate* identifiable regions or features mat make 

particular closed sampling path for scanning the mark. 

Fig 14 illustrates one example of a mark 308 mat has a. leas, an invariant ordtnal 
property, according .o one embodimen. of me invention. It should be appreciated, however, 
.h, marks having invariant ordinal as well as other topological properties according .o other 
embodiments of me invention are no. Hmited to the particular exemplary mark 308 shown m 
Fig ,4 The mark 308 includes three separate* identifiab.e differently colored regions 302 
fereen) 304 (red), and 306 (blue), respectively disposed with in a general mark area or spatial 
extent 309 Fig. 14 also shows an example of a scanning pa* 300 used » scan a, least a 

it falls within the mark area 309 when a portion of the image containing the mark 308 rs 
canned Whi.e the scanning path 300 is shown in Fig. !4 as an essentially circmar path, 
should be appreciated tha, the invention is no. limited in mis respec,; in particular, as 
discussed above, according ,0 o«her embodiment, me scanning pam 300 in Fig. 14 may be 
eimer an open line or curve or an essentially dosed pa.h <ha. fails wi«hin .he mark area 309 
when a portion of the image containing die mark 308 is scanned. 
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In Fig 14, the blue region 306 of the mark 308 is to the left of a line 3 1 0 between the 

egion 306 wi„ he on the left of the line 310 for any viewing angle (,e„ norma, or obhc,ue) 
,hemark308. According, one embooiment, the or.inal proper^ of *e mark 30S may be 
^nely detected by a scan along the scanning path 300 in either a Cockwise or counter- 
clockwise direction. 

order in which *e green region always preceded the blue region, the blue region always 
preceded the red region, and the red region always preceded the green region (e.g., green- 
b,u,red,b,ue-red-green,orre^^ 

pa^OOwouldresu.tinanorderinwhichgreena.waysp^edred.redaiwayspreceded 
blucandbluealwaysprecededgreen. .n one aspect of mis embodiment, the v^ous regtons 
of me mark 30S may be arranged such that for a grid of scanning pams mat are seauentrally 

path that passes through each of the regions of the mark 308. 

Another example of aninvarian. topological property ofamark according ,0 one 
embodiment of the invention is an "indusive property" of the mark. ,n particular, an 
.nc.usive property of a mark refers to a particular arrangement of a number of separate* 
identifiable regions or features tha, make up a mark, wherein a, .east one region or 
completely included within the spatia, extent of anomer region or feature. Similar to marks 
having an ordinal property, inclusive marks are particularly invariant a, leas, win, respect to 

viewing angle and scale of the mark. 

Fig ismustratesoneexampleofamarksntothasatleastaninvariantinclustve 

property, according to one embodiment of the invention. It should be appreciated, however 
mat marks having invariant inclusive as well as other topological properties accordmg to other 

Fig 15 The mark 3.2 includes three separately identifiable differently colored regtons 3,4 
(red) 3,6 (blue,, and 3,8 (green), respective,,, disposed within a mark area or spatia, extent 
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313. As illustrated in Fig. 15, the blue region 316 completely surrounds (i.e., includes) the red 
region 314, and the green region 318 completely surrounds the blue region 316 to form a 
multi-colored bulls-eye-like pattern. While not shown explicitly in Fig. 15, it should be 
appreciated that in other embodiments of inclusive marks according to the invention, the 
boundaries of the regions 314, 316, and 318 need not necessarily have a circular shape, nor do 
the regions 314, 316, and 318 need to be contiguous with a neighboring region of the mark. 
Additionally, while in the exemplary mark 312 the different regions are identifiable primarily 
by color, it should be appreciated that other attributes of the regions may be used for 
identification (e.g., shading or gray scale, texture or pixel density, different types of hatching 
such as diagonal lines or wavy lines, etc.) 

Marks having an inclusive property such as the mark 312 shown in Fig. 15 may not 
always lend themselves to detection methods employing a circular path (i.e., as shown in Fig. 
14 by the path 300) to scan portions of an image, as it may be difficult to ensure that the 
circular path intersects each region of the mark when the path is centered on the mark 
(discussed further below). However, given a variety of possible overall shapes for a mark 
having an inclusive property, as well as a variety of possible shapes (e.g., other than circular) 
for an essentially closed path or open line or curve path to scan a portion of an image, 
detection methods employing a variety of scanning paths other than circular paths may be 
suitable to detect the presence of an inclusive mark according to some embodiments of the 
invention. Additionally, as discussed above, other scanning methods employing point or area 
techniques may be suitable for detecting the presence of an inclusive mark. 

Yet another example of an invariant topological property of a mark according to one 
embodiment of the invention includes a region or feature count, or "cardinal property," of the 
mark. In particular, a cardinal property of a mark refers to a number N of separately 
identifiable regions or features that make up the mark which is invariant at least with respect 
to viewing angle. In one aspect, the separately identifiable regions or features of a mark 
having an invariant cardinal property are arranged with respect to each other such that each 
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region or feature is abie to be sampled in either an open line or curve or essentially closed 
path that lies entirely within the overall mark area (spatial extent) of tire mark as it appears m 

the image. 

In general, according to one embodiment, for marks that have one or bom of a cardtnal 
property and an ordinal property, the separately identifiable regions or features of the mark 
may be disposed with respect to each other such that when the mark is scanned in a scanmng 
path enclosing the center of the mark (e.g., an arcuate path, a spiral path, or a circular path 
centered on the mark and having a radius less man the radial dimension of the mark), tire path 
traverses a significant dimension (e.g., more man one pixel) of each separately identifiable 
region or feature of the mark. Furthermore, in one aspect, each of the regions or features of a 
mark having an invariant cardinal and/or ordinal property may have similar or identical 
geometric characteristics (e.g., size, shape); alternatively, in yet another aspect, two or more 
of such regions or features may have different distinct characteristics (e.g., different shapes 
and/or sizes). In this aspect, distinctions between various regions or features of such a mark 
may be exploited to encode information into the mark. For example, according to one 
embodiment, a mark having a particular unique identifying feature not shared with omer 
marks may be used in a reference target to distinguish the reference target ftom other targets 
ma, may be employed in an image metrology site survey, as discussed further below m 

Section I of the Detailed Description. 

Fig 16A illustrates one example of a mark 320 that is viewed normally and that has at 
.east an invariant cardinal property, according to one embodiment of the invention. It should 
be appreciated, however, that marks having invariant cardinal as well as other topologtcal 
properties according to other embodiments of the invention are no, limited to tire particular 
exemplary mark 320 shown in Fig. 16A. to mis embodiment, the maxk 320 includes a, least 
six separably identifiable dimensional regions 322A-322F (i.e., N = 6) that each 
emanates along a radial dimension 323 from a common area 324 (e.g., a center) of the mark 
320 in a spoke-like configuration. In Fig. 16A, a dashed-line perimeter outlines the mark area 
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321 (i.e., spatial extent) of the mark 320. While Fig. 16A shows six such regions having 
essentially identical shapes and sizes disposed essentially symmetrically throughout 360 
degrees about the common area 324, it should be appreciated that the invention is not limited 
in this respect; namely, in other embodiments, the mark may have a different number N of 
separately identifiable regions, two or more regions may have different shapes and/or sizes, 
and/or the regions may be disposed asymmetrically about the common area 324. 

In addition to the cardinal property of the exemplary mark 320 shown in Fig. 16A (i.e., 
the number N of separately identifiable regions), the mark 320 may be described in terms of 
the perimeter shapes of each of the regions 322A-322F and their relationship with one 
another. For example, as shown in Fig. 16 A, in one aspect of this embodiment, each region 
322A-322F has an essentially wedge-shaped perimeter and has a tapered end which is 
proximate to the common area 324. Additionally, in another aspect, the perimeter shapes of 
regions 322A-322F are capable of being collectively represented by a plurality of intersecting 
edges which intersect at the center or common area 324 of the mark. In particular, it may be 
observed in Fig. 16A that lines connecting points on opposite edges of opposing regions must 
intersect at the common area 324 of the mark 320. Specifically, as illustrated in Fig. 16A, 
starting from the point 328 indicated on the circular path 300 and proceeding counter- 
clockwise around the circular path, each edge of a wedge-shaped region of the mark 320 is 
successively labeled with a lower case letter, from a to /. It may be readily seen from Fig. 
16A that each of the lines connecting the edges a-g, b-h, c-i, d-j, etc., pass through the 
common area 324. This characteristic of the mark 320 is exploited in a detection algorithm 
according to one embodiment of the invention employing an "intersecting edges analysis," as 
discussed in greater detail in Section K of the Detailed Description. 

As discussed above, the invariant cardinal property of the mark 320 shown in Fig. 16A 
is the number N of the regions 320A-320F making up the mark (i.e., N = 6 in this example). 
More specifically, in this embodiment, the separately identifiable two-dimensional regions of 
the mark 320 are arranged to create alternating areas of different radiation luminance as the 
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mark is scanned along the — g path 300, shown for example in F ig . 16A as a 
path tha, is approximately centered around «he _ area 324. Stated different* - the 
mark is scanned along the scanning pa* m a significant dimension of each regron 322A- 
322 F is traversed to generate a scanned signal representing an alternating radiation lununance. 
At leas, one proper* of this aitemating radiation Krminance, name* atotal number 
of the radiation iuminance, is invariant a, .east with respect to viewing angie, as weil as 
changes of scaie (i.e., observation distance from the mark), in-plane regions of the mark, 
ligh ting conditions, arbitrary image content, etc., as discussed further below. 

Fig 16B is a graph showing a piot 326 of a iuminance curve (i.e., a scanned signai) 
^ is generated by scanning the mark 320 of Fig. i6A aiong me scanning path 300, starting 
ftom the point 328 shown in Fig. i6A and proceeding counter-ciockwise (a simiiar lununance 
patiemwouidresuitfromaciockwisescan). InFig. 16A, the .ighter areas between tire 
regions 322A-322F are respective* iabeied with encircled numbers i-6, and each corresponds 
,„ a respective successive half-cycie of higher iuminance shown in tire p.0.326 of Ftg. ,6B. 
,„ particuiar, for tire six region mark 320, the iuminance curve shown in Fig. KB has s.x 
cycles of alternating iuminance over a 360 degree scan around the path 300, as indicated ,n 
Fig . 16B by tire encircied numbers 1-6 corresponding to tire tighter areas between the regrons 

322A-322F of the mark 320. 

WhileFig. ,6Ashowsmemark320ates S entiall y anormalviewingangle,Fig. 17A 

shows the same mark 320 a, an oblige viewing angle of approximately 60 degrees off- 
norma!. Fig. 17B is a graph showing aplo, 330 of a iuminance curve (i.e., a scanned signai, 
that is generated by scanning tire obliquely imaged mark 320 of Pi, >7A along the scannmg 
path 300, in a manner similar to tha, discussed above in connection with Figs. 
From Fig 17B, it is still clear ma, there are six cycles of alternating luminance over a 360 

illustrated in Fig. 1 6B . 

Fig 18A shows the mark 320 again at essentially a normal viewing angle, but 



106 



-xod nf the mark 320 by an offset 362 between inc 
skewed off-center from the common area 324 of the marK . 

• fAP ^8 of the path 300 (discussed further below in 
common area 324 and a scanning center 338 of the pern; 

20. Fi g 18Bi S agraphshowing a plot332ofalummancecurve(,e.,a 

connection with Fxg. 20). F*. 18B P 18A along the skewed 

v»v tannine the mark 320 ot rig. i ^ 1a b 
scanned signal) that is generated by scanmngt ^^m-piss 16A, 

, oath 300 in a manner similar to that disused above m connects* F.gs. 
closed path 300, ma it is stiU clear that, almongh cycles are less 

16B.17A.-dim Aga,n, ft omF.g.l8B,.. I ss„l 
rcgu lar, there are six cycles of alternating luminance over a 360 degree scan 

pn , r0 dumber of cycles o t *el— ecurvegenemtedbyscannm gth mar, ng 
^ngpath300 ( either doctor is — ™* resp. Uo 

Ucircnlarpath) — and use any one or more of a variety of signa, recovery 

i L f y,hepre S ence ( orabsence,ofthemarUnanima 6 e U nderava,e t yof 1 ma g mg 
^'tcordingtooneembodimentofthe— , as discussed above, an automated 
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signal, wherein the scanning path is formed such that the scanning path falls entirely within 
the mark area if the scanned portion of the image contains the mark, and determining one of 
the presence and an absence of the mark in the scanned portion of the image from the scanned 
signal. In one aspect of this embodiment, the scanning path may be an essentially closed path. 
In another aspect of this embodiment, a number of different regions of a stored image are 
successively scanned, each in a respective scanning path to obtain a scanned signal. Each 
scanned signal is then respectively analyzed to determine either the presence or absence of a 
mark, as discussed further below and in greater detail in Section K of the Detailed 
Description. 

Fig. 19 is a diagram showing an image that contains six marks 320, through 320 6 , each 
mark similar to the mark 320 shown in Fig. 16A. In Fig. 19, a number of circular paths 300 
are also illustrated as white outlines superimposed on the image. In particular, a first group 
334 of circularpaths 300 is shown in a left-center region of the image of Fig. 19. More 
specifically, the first group 334 includes a portion of two horizontal scanning rows of circular 
paths, with some of the paths in one of the rows not shown so as to better visualize the paths. 
Similarly, a second group 336 of circular paths 300 is also shown in Fig. 19 as white outlines 
superimposed over the mark 320 5 in the bottom-center region of the image. From the second 
group 336 of paths 300, it may be appreciated that the common area or center 324 of the mark 
320 5 falls within a number of the paths 300 of the second group 336. 

According to one embodiment, a stored digital image containing one or more marks 
may be successively scanned over a plurality of different regions using a number of respective 
circular paths 300. For example, with the aid of Fig. 19, it may be appreciated that according 
to one embodiment, the stored image may be scanned using a number of circular paths, 
starting at the top left-hand corner of the image, proceeding horizontally to the right until the 
right-most extent of the stored image, and then moving down one row and continuing the scan 
from either left to right or right to left. In this manner, a number of successive rows of 
circular paths may be used to scan through an entire image to determine the presence or 
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abs enceofa mark in each region. In generai, ? should be appreciate, tha, a variety o 

to determine the presence or absence of marks in the image. 

F „r purposes of this disclosure, a "scanning cente^ is a point nr an mrage to be tested 

, of a mark In one embodiment of the invention as shown m Fig. 19, a 

for the presence ol a marK. uiuuccu 

Lgasca^ngcen tt r33S.Inu 1 ee X amp 1 eofFig.20,U8pi X e 1 seachhavmgarad^ 

* are shown for purpose, of iUustiation only, and that the invention is no, hmrted to 

example shown in Fig. 20. ' 

In particular, according to one embodiment of the invention, a radtus 339 of tire 
circular path 300 from tire scanning center 338 is a parameter tha, may be predetermrned 
Id) or ad j ustab 1 emadetec,io„a, g ori,nm according, one embodiment of —o, 

.eovera.Upatiale.entoftiremarkorm^.obede^dintheimage.Fore^e,^ 
d to e»sion323is.i k ewiseindica te dfor«emark3 2 0 s inFig.l9. According to one 

32 3 From the foregoing, h shou,d be appreciated that the range of posstble radn 339 
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V arious P aths300,— 

300(e g ,asshow.in« 

!• • ofthemariOasitisexpectedtoappearinanimage.Inpart.cular^na 

radial dimension of the mark) as v j: 11c iiq of a given 

• , -thm according to one embodiment of the invention, the radius 339 of gw 
detection algorithm according 10 o <, it5tanC es 
• o nath 300 may be adjusted to account for various observation distances 
circular scanning path 3UU may dc j 

♦ • «a the mark and a camera obtaining an image of the scene, 
between a scene containing the marK anu a ^ 

,■ a .mole 344 (ti which indicates a rotation from a 
Fie 20 also illustrates a sampling angle 344 W,w 

■ t, „ thestartingpoint328showninFig.20)ofaparticularpixel 

scanning reference point (e.g., the starting po 

!M alone the path 300. Accordingly, it should be appreciated that the sampling 

being sampled along trie paxn 

,„ «n deerees for each scan along a circular path 300. rig. 
angle granges from zero degrees to 360 degrees tor 

2, is a graph of a plot 342 showing the sampling angle * (on the vertical axts of the graph) 
1^21 it may be seen that, due to the discrete pixel nature of the scanned image, the 

--^r— rinrr 

(i e th eplo,342isnotas,raightlinehe W e,n Z c r odegreesand360 d egrees). g , 

Verses each separately identifiable region or feature of a mar, (i.e., one or - of the 
lofthema* similarto nta, shown in Fig, !6B, 17B, and 18B. Alternatively, as 

arbitrary contents of the image in the scanned reg.on. For example, 

f „hitP naner having an uneven surface (e.g., the region 
region of an image of white paper naving 
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firs. group334ofpafts shown inFig. 19). As discussed ftrther below, it may be apprecated 
from Fig. 22B fta, a particular number of cycles is no. evident in .he random signal. 

As can be seen, however, from a comparison of me luminance curves shown m F.gs. 
16B 17B, and UB. in which a particular number of cycles is cviden. in the curves, bom me 
viewing angle and translation of the mark 320 relative «o fte circular path 300 affects me 
"uniformUy of me .uminance curve. For purposes of .his disclosure, the term -uniform,^ 
refers to the constancy or regular^ of a process «ha. genera.es a signal which may include 
some noise s,atistics. One example of a uniform signa! is a sine wave having a constant 
frequency and amplitude. In view of fte foregoing, i. can be seen from Fig. 16B fta. me 
.uminance curve chained by circuiarly scanning fte normally viewed mark 320 shown in F,g. 
,6A (i.e., when me paft 300 is essentially centered about fte common area 324) is essentra.lv 
uniform, as a period 334 between two consecutive peaks of the luminance curve is 
approxima.elyftesameforeachpairofpeaksshownmFig.16B. 

curve of Fig 17B (obtained by circularly scanning the mark 320 at an oblique viewing angle 
of approximately 60 degrees) as well as the .uminance curve of Fig. 1«B (where fte pa* 300 
is skewed off-cen.er from fte common area 324 of fte mark by an offset 362) is non-uniform, 
as the regularity of the circular scanning process is disrupted by .he rotation or the translation 
of fte mark 320 with respect to the paft 300. 

Regardless of the uniformity of the luminance curves shown in Figs. 16B, 17B, and 
1 8B however, as discussed above, i. should be appreciated that a signal having a known 
invariant number ofcyc.es based on fte cardinal property of a mark can be recovered from a 
variety of luminance curves which may indicate translation and/or roration of fte mark; m 
particular, several conventional methods are known for de.ec.ing bom uniform sig„a.s and 
non-uniform signals in noise. Conventional signa. recovery meftods may emp.oy vanous 
processing techniques induding, but not lhni.ed,o, Kalman fihering, short-time Founer 
U^sform, parametric model-based detection, and cumulative phase rotation analysis, some of 
which are discussed in greater detail below. 
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One method that may be empioyed by detection algorithms according to various 
embodiments of the present invention for processing either uniform or non-uniform signals 
involves detecting an instantaneous phase of the signal. This method is commonly referred to 

i % a „j JcHi ^rn^ed in greater detail in Section K of the 
as cumulative phase rotation analysis and is discussed in grea 

Detailed Description. Figs. 16C, 1TC 1 8C are graphs showing respective plots 346, 348 and 
350 of a cumulative phase rotation for the luminance curves shown in Figs. 16B, !7B and 
18B respectively. Similarly, Fig. 22C is a graph showing aplo. 366 of a cumulative phase 
rotation for the luminance curve shown in Fig. 22B (i.e., representing a signal generated from 
a scan of an arbitrary region of an image tha, does no. include a mark). According to one 
embodiment of the invention discussed further be.ow, the non-uniform signals of Figs. 17B 
and 1 SB may be particularly processed, for example using cumulative phase rotation analysis, 
,o not only detect the presence of a mark but to also derive the offset (skew or translation) 

1 \ e^ m »rk Hence valuable information may be obtained 
and/or rotation (viewing angle) of the mark. Hence, vaiu 

from such non-uniform signals. ^ 
Given a mark having N separately identifiable features symmetrically disposed around 
a center of the mark and scanned by a circular pad, centered on the mark, the ins_us 
cumulative phase rotation of a perfectly uniform; luminance curve (i.e., no rotation or 
^slationofthe mark with respect to tire circular path) is given by N* as the circular path is 
traversed, where * is tine sampling angle discussed above in connection with Fig, 20 and 21 . 
Witir respect to the mark 320 in which N = 6, a reference cumulative phase rotation based on a 
perfectly uniform luminance curve having a frecmency of 6 cyc.es/scan is given by 6*. as 
shownby tire straight line 349 indicated in each of Figs. 16C, 17C, 18C, and 22C. 
Accordingly, for a maximum sampling angle of 360 degrees, the maximum cumulative phase 
rotation of the luminance curves shown inFigs. 16B, 17B, and 18B is 6 x 360 degrees - 2,60 
degrees. 

For example, the luminance curve of Fig. 16B is approximately a stationary sine wave 
that completes six 360 degree signal cycles. Accordingly, the plot 346 of Fig. 16C 
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presenting the cumulative phase rotation of the luminance curve of Fig. 16B shows a 
relatively steady progression, or phase accumulation, as the circular pa* is traversed, ieadmg 
to a maximum of 2160 degrees, with relatively minor deviations from the reference 

cumulative phase rotation line 349. 

Similarly, the luminance curve shown in Fig. 17B includes six 360 degree signal 
cycles; however, due to the 60 degree oblique viewing angle of the mark 320 show* in Fig. 
,7A the luminance curve of Fig. 17B is no, uniform. As a result, mis signa. non-uniformtty 
is reHected inthe plot 348 of the cumulative phase rotation shown in Fig. 17C, which is no. a 
smooth, steady progression leading to 2016 degrees. In particular, the plot 348 deviates from 
the reference cumulative phase rotation Une 349, and shows two distinct cycles 352A and 
352B relative to the line 349. These two cycles 352A and 352B correspond to the cycies m 
Fig 17B where the regions of the mark are foreshortened by the perspective of the obhque 
viewing angle. In particular, in Fig. 17B, the cycle labeled with tire encircled number 1 rs 
wide and hence phase accumulates more slowly than in a uniform signal, as indicated by the 
encircled number I in Fig. 17C. This initial wide cycle is followed by two narrower cycles 2 
and 3 for which the phase accumulates more rapidly. This sequence of cycies is followed by 
another pattern of a wide cycle 4, followed by two narrow cycles 5 and 6, as indicated in both 

of Figs. 17B and 17C. 

The luminance curve shown in Fig. 1 8B also includes six 360 degree signal cycles, 
ar.d so again the total cumulative phase rotation shown in Fig. 18C is a maximum of 2160 
degrees. However, as discussed above, the luminance curve of Fig. 18B is also non-uniform, 
similar ,o tha, of the curve shown in Fig. 17B, because the circular scanning path 300 shown 
in Fig 18A is skewed off-center by the offset 362. Accordingly, the plot 350 of the 
cumulative phase rotation shown in Fig. 18C also deviates from the reference cumulative 
phase rotation line 349. In particular, the cummative phase rotation shown in F.g. 18C 
includes one haif-cycle of lower phase accumulation followed by one half-cycle of htgher 
phase accumulation relative to the line 349. This cycle of lower-higher phase accumulation 
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. • i ath^OO As a mark is viewed obliquely (asm Figs. 17A,17B, and 
renter" by the circular path 3UU. AsanuuK». 

deviation of the measured — e phase rotation (i.e., .he pio, 366 m Fig. 22C) 
scanned signa, fro. the reference — e phase rotation iine 349 is significant as 

aJenceo^e^in^esoan.P— , aceo^ «o one aspec, <> f «s e— , 
indicated by period-two and period-one ^respective*, that ate present inthe 

cumuiative phase rotation iine 349. TTte mathematica, of a detection 

employing a cumuiative phase rotation analysis according to one embodiment of the 

According to one embodiment of me invention, a detection aigorithm empioymg 
image to identify one or more candidates for the presence of a ma* in the image. 



114 



However, i, is possible that one or more fa.se positive — es may be identifiedm - 
htitial pass through ,he image. In parties, the number of fa.se positives identified by the 
^gorithm may be based in par, on ,he se.ee.ed radius 339 of ,he circuiar path 300 (e.g., *e 
Fig 20) with respect to the overa.. size or spatia. extent of the mark being sought (e.g., the 
radial dimension 323 ofthe mark 320). According to one aspect of mis embodiment, 
however, it may be desirab.e » seiect a radius 339 for the circular path 300 such that no vahd 
candidate be rejected in an initial pass through the image, even though fa.se positives may be 
identic, in genera., as discussed above, in one aspect the radius 339 shou.d be smaU 
enough relive to the apparent radius ofthe image of the mark to ensure that a, .east one of 
the paths lies entirely within the mark and encircles the center of the mark. 

Once a detection algorithm initially identifies a candidate mark in an image (e.g., 
based on either a cardinal property, an ordinal property, or an inclusive property of the mark, 
as discussed above), the detection algorithm can subsequently indude a refinement process 
ma, further tests other properties of me mark tha, may no, have been initially tested, usurg 
ahemative detection aigorithms. Some alternative detection algorimms according to other 
embodiments ofthe invention, mat may be used either a.o„e or in various combinations wrth a 
cumulative ptase rotation analysis, are discussed in detail in Section K of the Detailed 
Description. 

With respect to detection refinement, for example, based on the cardinal property of 
me mark 320, some geometric properties of symmetrically opposed regions of the mark are 
similarly affeeted by translation and rotation. This phenomenon may be seen, for example, m 
Fig 17A, in which me upper and lower regions 322B and 322E are distorted due to the 
oblique viewing angle to be long and narrow, whereas the upper left region 322C and the 
,„wer right region 322F are distorted to be shorter and wide, According to one embodmten, 
by comparing the geometric properties of area, major and minor axis lengm, and orientatio n 
of opposed regions (e.g., using a "regions analysis" method discussed in Section K ofthe 
Deta i,ed Description), many candidate marks mat resemble the mark 320 and ma, are false,. 
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identified in a first pass through the image may be eliminated. 

Additionally, a particular artwork sample having a number of marks may have one or 
more properties that may be exploited to rule out false positive indications. For example, as 
shown in Fig. 16A and discussed above, the arrangement of the separately identifiable regions 
of the mark 320 is such that opposite edges of opposed regions are aligned and may be 
represented by lines that intersect in the center or common area 324 of the mark. As 
discussed in greater detail in Section K of the Detailed Description, a detection algorithm 
employing an "intersecting edges" analysis exploiting this characteristic may be used alone, or 
in combination with one or both of regions analysis or cumulative phase rotation analysis, to 
refine detection of the presence of one or more such marks in an image. 

Similar refinement techniques may be employed for marks having ordinal and 
inclusive properties as well. In particular, as a further example of detection algorithm 
refinement considering a mark having an ordinal property such as the mark 308 shown in Fig. 
14, the different colored regions 302, 304 and 306 of the mark 308, according to one 
embodiment of the invention, may be designed to also have translation and/or rotation 
invariant properties in addition to the ordinal property of color order. These additional 
properties can include, for example, relative area and orientation. Similarly, with respect to a 
mark having an inclusive property such as the mark 312 shown in Fig. 15, the various regions 
314, 316 and 318 of the mark 312 could be designed to have additional translation and/or 
rotation invariant properties such as relative area and orientation. In each of these cases, the 
property which can be evaluated by the detection algorithm most economically may be used 
to reduce the number of candidates which are then considered by progressively more intensive 
computational methods. In some cases, the properties evaluated also can be used to improve 
an estimate of a center location of an identified mark in an image. 

While the foregoing discussion has focussed primarily on the exemplary mark 320 
shown in Fig. 16A and detection algorithms suitable for detecting such a mark, it should be 
appreciated that a variety of other types of marks may be suitable for use in an image 
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metrology reference targe, (similar to «he Urge, 120A shown in Fig. 8), according to other 
embodiments of the invention (e.g., marks having an ordinal property similar to the mark 308 
shown in Fig. 14, marks having an inclusive property similar to the mark 312 shown in F.g. 
15 etc) Inparticular.Figs^SAandaSBshowyetanotherexampleofarobustmarkSeS 
according to one embodiment of the invention that incorporates bo* cardinal and ordinal 
properties. 

The mark 368 shown in Fig. 23A utilizes at least two primary colors in an arrangement 
of wedge-shaped regions similar to ma, shown in Fig. 16A for the mark 320. Specifically, in 
one aspec, of tins embodiment <he mark 368 uses to the primary colors blue and yellow » a 
repeating pattern of wedge-shaped regions. Fig. 23A shows a number of black colored 
regions 320A, each followed in a counter-clockwise order by a blue colored region 370B, a 
green colored region 370C (a combination of blue and yellow), and a yellow colored reg,on 
370D Fig. 23B shows the image of Fig. 23 A filtered to pass only blue light. Hence, m F.g. 
23B me "clear" regions 370E between two darker regions represent a combination of the blue 
and green regions 370B and 370C of the mark 368, while the darker regions represent a 
combination of the black and yellow regions 370A and 370D of the mark 368. An image 
similar to that shown in Fig. 23B, although rotated, is obtained by filtering the image of F,g. 
23A to show only yellow light. The two primary colors used in the mark 368 establish 
quadrature on a color plane, from which it is possible to directly generate a cumulative phase 
rotation, as discussed further in Section K of the Detailed Description. 

Additionally, Fig. 24A shows yet another example of a mark suitable for some 
embodiments of the present invention as a cross-hair mark 358 which, in one embodiment 
may be used in place of any one or more of the asterisks serving as the fiducial marks 124A- 
124D in the example of tire reference targe, 120A shown in Fig. 8. Additionally, accordmg ,o 
one embodiment the example of the inclusive rnark 3.2 shown in Fig. 15 need no, necessanly 
include a number of respective differently colored regions, but instead may include a number 
of alternating co.ored, black and white regions, or differently shaded and/or hatched region, 
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From the foregoing, it should be appreciated tha, a wide variety of landmarks for machine 
vision in general, and in particular fiducial marks for image metrology applications, are 
provided according to various embodiments of the present invention. 

According to another embodiment of the invention, a landmark or fiducial mark 
according to any of the foregoing embodiments discussed above may be printed on or 
Chemise coupled ,0 a substrate (e.g., tire substrate .33 of the reference targe, 120A shown » 
Figs 8 and 9). In particular, in one aspect of Oris embodiment, a landmark or fiducal mark 
according ,0 any of the foregoing embodiments may be printed on or otherwise coupled to a 
self-adhesive substrate that can be affixed to an object. For example, Fig. 24B shows a 
substrate 354 having a self-adhesive surface 356 (i.e., a rear surface), on which is printed (,e„ 
on a front surface) the mark 320 of Fig. 16A. In one aspect, the substrate 354 of Fig. 24B 
may be a self-stick removable note tha. is easily affixed a. a desired location in a scene pnor 
,„ obtaining one or more images of the scene ,0 .facilitate automatic feature detection. 

In particular, according to one embodiment, marks printed on self-adhesive substrates 
may be affixed at desired locations in a scene * facilitate automatic identification of objects 
of interest in the scene for which position and/or size information is not known but desrred. 
Additionally, such self-stick notes including prints of marks, according to one embodtment of 
.he invention, may be placed in the scene a. particular locations to esK.bl.sh a relationship 
between one or more measuremen. planes and a reference plane (e.g., as discussed above m 
Section C of tire Detailed Description in connection with Fig. 5). In ye. another embodiment, 
such self-stick notes may be used to facilitate automatic detection of link points be<wee„ 
multiple images of alarge and/or complex space, for purposes of site surveying using image 
metrology methods and apparatus according .o me invention. In ye. another embodiment, a 
plurality of uniquely identifiable marks each printed on a self-adhesive substrate may be 
placed in a scene as a plurality of objects of interest for purposes of facilitating an au,oma.,c 
multiple-image bundle adjustmen. process (as discussed above in Section H of .he Descripnon 
of me Related Ar,), wherein each mark has a uniqudy identifiable physical attribute that 
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aUows for automatic "referencing" of tne ma* in a number of images. Such an automatic 



nficantly reduces the probability of analyst blunders that may 



occur 

r. A i j.,^^r. +h<* nrnnanuuv ui cxiiai y ^*v**«. 

referencing process sigm 



These and other exemplary applications for "self-stick 



during a manual referencing process 
landmarks" or "self-stick fiducial marks" are discussed tether below in Section I of the 



Detailed Description. 



H. Exemplary Image Processing Methods for Image Metrology 
According to one embodiment of the invention, the image metrology processor 36 of 
Fig 6 and the image metrology server 36A of Fig. 7 taction similarly (i.e., may perform 
similar methods) with respect to image processing for avariety of image metrology 
appiications. Additionally, according ,0 one embodiment, one or more image metro.ogy 
S ervers similar to the image metrology server 36A shown in Fig. 7, as we,, as the various 
Cent processors 44 shown in Fig. 7, may perform various image metroiogy methods ,n a 
distributed manner; in partial as discussed above, some of the functions described herem 
with respect to image metro,ogy methods may be performed by one or more image metroiogy 
servers while other functions of such image metroiogy methods may be performed by one or 
more client processors 44. ,n Ms manner, in one aspect, various image metrology methods 
according to the invention may be implemented in a modular manner, and executed » a 
distributed fashion amongst a dumber of different processors. 

FoUowing below is a discussion of exemplary automated image processing methods 
for image metrology applications according to various embodiments of the invention. The 
material in this section is discussed in greater detail (including several matanafcal 
derivations) in Section L of the Detailed Description. Although the discussion be.ow focuses 
on automated image processing methods based in par, on some of the novel machine vis.on 
techniques discussed above in Sections 03 and K of the Detailed Description, it shou,d be 
apprecia^dthatsuchimageprocessingmemods may be modiHed ,„ aUow for various leveis 
of user interaction if desired for a particular application (e.g., manual rather man automatic 
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identification of one or more reference targe* or control points in a scene, manna, rather than 
automatic identification of object points of interest in . scene, manna, rate than automatic 
identification of multi-image link points or various measurement p,anes with respect to a 
reference p.ane for the scene, etc.). A number of exempt implementations for the image 
metrology methods discussed herein, as well as various image metro.ogy apparatus accordmg 
to the invention, are discussed further in Section I of the Detailed Description. 

According to one embodiment, an image metrology method first determines an nuUa! 
estimate of a, least some camera calibration information. For example, the method may 
determine an initial estimate of camera exterior orientation based on assumed or estimated 
interior orientation parameters of the camera and reference information (e.g„ a particular 
artwork model) associated with a reference target placed in the scene, .n this embodiment, 
based on these initial estimates of camera calibration information, a least-squares iterative 
algorithm subsequently is employed to refine ft. estimates. In one aspect, the only 
requirement of the initial estimation is that it is sufficiently close to the tine solution so tha, 
tire iterative algorithm converges. Such an estimation/refinement procedure may be 
performed using a single image of a scene obtained a. e*ch of one or more different camera 
iocations to obtain accurate camera calibration information for each camera location. 
Subsequently, this camera caUbration information may be used to determine actual posttion 
and/or size information associated with one or more objects of interest in the scene tha, are 
identified in one or more images of the scene. 

Figs 25A and 25B illnstiate a flow chart for an image metrology method according to 
one embodiment of the invention. As discussed above, the method outtined in Figs. 25A and 
25B is discussed in greater detail in Section L of the Detailed Description. It should be 
appreciated tha, the method of Fig, 25A and 25B provides merely one example of .mage 
processing for image metroiogy applications, and that the invention is no, limited to tins 
particular exemplary method. Some examples of alternative methods and/or alternative steps 
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for the methods of Fig, 25A and 25B a* also discussed beiow and in Section L of the 
Detailed Description. 

The method of Fig, 25A and 25B is described beiow, for purposes of illustratton, wth 
referenee to the image metro.ogy apparatus shown in Fig. 6. As discusseu above, it should be 
appreciated 4a. the meftod of Figs. 25A and 25B simiiarly may be performed usmg the 
various image metrology apparatus shown in Fig. 7 (i.e., network implementahon). 

With reference to Fig. 6, in block 502 of Fig. 25A, a user enters or downloads to the 
processor 36, via one or more user interfaces (e.g., the mouse 40A and/or keyboard 40B), ^ 
e^era mode, estimates or manufacturer data for the camera 22 used to obtain an image 20B 
of the scene 20A. As discussed above in Section E of the Description of theRelated Art, the 
camera mode. generaUy includes interior orientation parameters of the camera, such as the 
principal distance for a particuiar focus setting, the respective x- and y- coordinates m the 
ira age plane 24 of the principa, point (i.e., the point a. which the optica! axis S2 of the camera 
actually intersects the image plane 24 as shown in Fig. 1), and the aspect ratio of ore CCD 
array of the camera. Additionally, the camera model may inciude one or more parameters 
relating ,0 lens distortion effects. Some or all of these camera model parameters may be 
provided by. the manufacturer of the camera and/or may b. reasonably estimated by the user. 
For example, the user may enter an estimated principal distance based on a particular focal 
setting of the camera a, the time the image 20B is obtained, and may also initially assume that 
the aspect rauo is eoual to one, mat the principal point is a, the origin of the image plane 24 
(see, for example, Fig. 1), and tha, there is no significant lens distortion (e.g., each !ens 
diaortion parameter, for example as discussed above in connection with Eo, (8), ts se, to 
K ro) It should be appreciated that the camera model estimates or manufacturer data may be 
manually entered to the processor by the user or downloaded to the processor, for example, 
from any one of avariety of portable storage media on which the cameramodei data is stored. 

I„ block 504 of Fig. 25A, the user enters or downloads to the processor 36 (e.g., vra 
one or more of the user interfaces) the reference information associated with the reference 
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target 120A (or any of a variety of other reference targets according to other embodiments of 
the invention). In particular, as discussed above in Section Gl of the Detailed Description in ' 
connection with Fig. 10, in one embodiment, target-specific reference information associated 
with a particular reference target may be downloaded to the image metrology processor 36 
using an automated coding scheme (e.g., a bar code affixed to the reference target, wherein 
the bar code includes the target-specific reference information itself, or a serial number that 
uniquely identifies the reference target, etc.). 

It should be appreciated that the method steps outlined in blocks 502 and 504 of Fig. 
25 A need not necessarily be performed for every image processed. For example, once camera 
model data for a particular camera and reference target information for a particular reference 
target is made available to the image metrology processor 36, that particular camera and 
reference target may be used to obtain a number of images that may be processed as discussed 
below. 

In block 506 of Fig. 25A, the image 20B of the scene' 20 A shown in Fig. 6 (including 
the reference target 120A) is obtained by the camera 22 and downloaded to the processor 36. 
In one aspect, as shown in Fig. 6, the image 20B includes a variety of other image content of 
interest from the scene in addition to the image 120B of the reference target (and the fiducial 
marks thereon). As discussed above in connection with Fig. 6, the camera 22 may be any of a 
variety of image recording devices, such as metric or non-metric cameras, film or digital 
cameras, video cameras, digital scanners, and the like. Once the image is downloaded to the 
processor, in block 508 of Fig. 25A the image 20B is scanned to automatically locate at least 
one fiducial mark of the reference target (e.g., the fiducial marks 124A-124D of Fig. 8 or the 
fiducial marks 402A-402D of Fig. 10B), and hence locate the image 120B of the reference 
target. A number of exemplary fiducial marks and exemplary methods for detecting such 
marks are discussed in Sections G3 and K of the Detailed Description. 

In block 510 of Fig. 25 A, the image 120B of the reference target 120A is fit to an 
artwork model of the reference target based on the reference information. Once the image of 
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the reference target is reconciled with the artwork model for the target, the ODRs of the 
reference target (e.g., the ODRs 122A and 122B of Fig. 8 or the ODRs 404A and 404B of Fig. 
10B) may be located in the image. Once the ODRs are located, the method proceeds to block 
512, in which the radiation patterns emanated by each ODR of the reference target are 
analyzed. In particular, as discussed in detail in Section L of the Detailed Description, in one 
embodiment, two-dimensional image regions are determined for each ODR of the reference 
target, and the ODR radiation pattern in the two-dimensional region is projected onto the 
longitudinal or primary axis of the ODR and accumulated so as to obtain a waveform of the 
observed orientation dependent radiation similar to that shown, for example, in Figs. 13D and 
Fig. 34. In blocks 514 and 516 of Fig. 25A, the rotation angle of each ODR in the reference 
target is determined from the analyzed ODR radiation, as discussed in detail in Sections J and 
L of the Detailed Description. Similarly, according to one embodiment, the near-field effect 
of one or more ODRs of the reference target may also be exploited to determine a distance 
Zcam between the camera and the reference target (e.g., see Fig. 36) from the observed ODR 
radiation, as discussed in detail in Section J of the Detailed Description. 

In block 518 of Fig. 25A, the camera bearing angles a 2 and y 2 (e.g., see Fig. 9) are 
calculated from the ODR rotation angles that were determined in block 514. The relationship 
between the camera bearing angles and the ODR rotation angles is discussed in detail in 
Section L of the Detailed Description. In particular, according to one embodiment, the camera 
bearing angles define an intermediate link frame between the reference coordinate system for 
the scene and the camera coordinate system. The intermediate link frame facilitates an initial 
estimation of the camera exterior orientation based on the camera bearing angles, as discussed 
further below. 

After the block 518 of Fig. 25A, the method proceeds to block 520 of Fig. 25B. In 
block 520, an initial estimate of the camera exterior orientation parameters is determined 
based on the camera bearing angles, the camera model estimates (e.g., interior orientation and 
lens distortion parameters), and the reference information associated with at least two fiducial 
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marks of .he reference ..get In par.icu.ar, in Hoc, 520, tire reia.ionship between *. camera 
coordina.e system and .he in,— in* toe is established using .he camera beanng 
^es and .he reference information associated witi, a. leas, two fiducial marks .0 soive a 

Unk frame is known, an initial c,ima.e of me camera exterior orient may be obtatned by 
a series of transformations from tire reference coordina,e sys.em .o Ore .ink frame, the hnk 
.tame .o 4. camera coordina te system, and tire camera coordinare sys.em .o me image pianc 

of the camera. 

Once an initial estimate of camera exterior oriemation is determined, biock 522 of 
Fig 25B indices ma. es.ima.es of camera calibration information in genera! (e.g., m.enor 
and exterior orientation, as weii as lens dis.or.i0n parameters) may be refuted by ieas.-so.uares 
iteration. In particuiar, in biock 522, one or more of the initiai estimation 
oriemation from biock 520, any camera mode, estimafcs from block 502, me reference 
information from block 504, and .he disfcnce z„ from block 5 ,6 may be used as inpu. 
paramCers .o an irerative leas,-s q uares a,gori,hm (discussed in de«ai> in Section L of me 
Betailed Description, .o obtain a cor»ple.e coordina.e sys.em transformation from the camera 
im age plane 24 .o tire reference coordina,e sysKm 74 for me scene (as shown, for example, .n 
Figs 1 or 6, and as discussed above in connection with Eq. (1 1) )• 

' ,„ block 524 of Fig. 25B, one or more points or objects of inieres. in me s«ne for 
which position and/or size information is desired are manually or automatically identified 
fromtheimageofthe scene. For examp,e, as discussed above in Section C of tite D eta,,ed 
Description and in connection with Fig. 6, a user may use one or more user interfaces .o select 
(e g via point and click using a mouse, or a cursor movement) various features of m<ere, 

in tire scene may he automatical identified by attaching .o such objects one or more robus. 
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fiducial mark, (RFIDs) (e.g., using self-adhesive removable notes having one or more RFIDs 
printed .hereon), as discussed further below in Section I of the Defiled Description. 

In block 526 of Fig. 25B, the method queries if the points or objects of tnterest 
identified in Ore image He in the reference plane of the scene (e.g., the reference plane 2! of 
the scene 20A shown in Fig. 6). .f such points of interest do no, lie in the reference plane, the 
method proceeds to block 528, in which the user enters or downloads to the processor the 
relationship or transformation between the reference plane and a measurement plane in whtch 
the points of interest lie. For example, as illustrated in Fig. 5, a measurement plane 23 » 
which points or objects of interest lie may have any known arbitrary relationship » the 
reference plane 21 . In particular, for built or planar spaces, a number of measurement planes 
may be selected involving 90 degree transformations between a given measurement plane and 

the reference plane for the scene. 

,n block 530 of Fig. 25B, once it is determined whether or no. the points or objec* of 
interest lie in me reference plane, the appropriate coordinate system reformation may be 
applied to the identified points or objects of interest (e.g., either a transformation between the 
camera image plane and tire reference plane or the camera image plane and the measurement 
plane) to obtain position and/or size information associated with tire points or objects of 
interest. As shown in Fig. 6, such position and/or size information may include, but , no, 
limited to, a physical distance 30 between «wo indicated poin,s 26A and 28A in tire scene 
20A. 

In tire image metrology method outlined in Figs. 25A and 25B, it should be 
appreciated that other alternative steps for the method to determine an initial estimation of the 
camera exterior orientation parameters, as se, forth in blocks 510-520, are possible. In 
particular, according ,o one alternative embodiment, an initial estimation of the exterior 
orientation may be determined solely from a number of fiducial marks of the reference target 
without necessarily using data obtained from one or more ODRs of tire reference target. For 
example, reference target orientation (e.g., pitch and yaw) in the image, and hence camera 
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bearing, may be estimated from cumulative phase rotation curves (e.g., shown in Figs. 16C, 
17C, and 18C) generated by scanning a fiducial mark in the image, based on a period-two 
signal representing mark tilt that is present in the cumulative phase rotation curves, as 
discussed in detail in Sections G3 and K of the Detailed Description. Subsequently, initial 
estimates of exterior orientation made in this manner, taken alone or in combination with 
actual camera bearing data determined from the ODR radiation patterns, may be used in a 
least squares iterative algorithm to refine estimates of various camera calibration information. 

/. Exemplary Multiple-Image Implementations 

This section discusses a number of exemplary multiple-image implementations of 
image metrology methods and apparatus according to the invention. The implementations 
discussed below may be appropriate for any one or more of the various image metrology 
applications discussed above (e.g., see Sections D and F of the Detailed Description), but are 
not limited to these applications. Additionally, the multiple-image implementations discussed 
below may involve and/or build upon one or more of the various concepts discussed above, 
for example, in connection with single-image processing techniques, automatic feature 
detection techniques, various types of reference objects according to the invention (e.g., see 
Sections B, C, G, Gl, G2, and G3 of the Detailed Description), and may incorporate some or 
all of the techniques discussed above in Section H of the Detailed Description, particularly in 
connection with the determination of various camera calibration information. Moreover, in 
one aspect, the multiple-image implementations discussed below may be realized using image 
metrology methods and apparatus in a network configuration, as discussed above in Section E 

of the Detailed Description. 

Four exemplary multi-image implementations are presented below for purposes of 
illustration, namely: 1) processing multiple images of a scene that are obtained from different 
camera locations to corroborate measurements and increase accuracy; 2) processing a series of 
similar images of a scene that are obtained from a single camera location, wherein the images 
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have consecutively larger scales (i.e. the images contain consecutively larger portions of the 
scene), and camera calibration information is interpolated (rather than extrapolated) from 
smaller-scale images to larger-scale images; 3) processing multiple images of a scene to 
obtain three-dimensional information about objects of interest in the scene (e.g., based on an 
automated intersection or bundle adjustment process); and 4) processing multiple different 
images, wherein each image contains some shared image content with another image, and 
automatically linking the images together to form a site survey of a space that may be too 
large to capture in a single image. It should be appreciated that various multiple image 
implementations of the present invention are not limited to these examples, and that other 
implementations are possible, some of which m*y be based on various combinations of 
features included in these examples. 

Processing Multiple Images to Corroborate Measurements and 
Increase Accuracy 

According to one embodiment of the invention, a number of images of a scene mat are 
obtained from different camera locations may be processed to corroborate measurements 
and/or increase the accuracy and reliability of measurements made using the images. For 
example, with reference again to Fig. 6, two different images of the scene 20A may be 
obtained using the camera 22 from two different locations, wherein each image includes an 
image of the reference target 120A. In one aspect of this embodiment, the processor 36 
simultaneously may display bom images of the scene on the display 38 (e.g. using a split 
screen), and calculates the exterior orientation of me camera for each image (e.g., accordmg to 
the method outlined in Figs. 25A and 25B as discussed in Section H of the Detailed 
Description). Subsequently, a user may identify points of interest in the scene via one of the 
displayed images (or points of interest may be automatically identified, for example, using 
stand-alone RFlDs placed at desired locations in the scene) and obtain position and/or s,ze 
information associated with the points of interest based on the exterior orientation of the 
camera for the selected image. Thereafter, tire user may identify me same points of interest in 
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the scene via another of the displayed images and obtain position and/or size information 
based on the exterior orientation of the camera for this other image. If the measurements do 
not precisely corroborate each other, an average of the measurements may be taken. 

12. Scale-up Measurements 
According to one aspect of the invention, various measurements in a scene may be 
accurately made using image metrology methods and apparatus according to at least one 
embodiment described herein by processing images in which a reference target is 
approximately one-tenth or greater of the area of the scene obtained in the image (e.g., with 
reference again to Fig. 6, the reference target 120A would be approximately at least one-tenth 
the area of the scene 20A obtained in the image 20B). In these cases, various camera 
calibration information is determined by observing the reference target in the image and 
knowing a priori the reference information associated with the reference target (e.g., as 
discussed above in Section H of the Detailed Description). The camera calibration 
information determined from the reference target is then extrapolated throughout the rest of 
the image and applied to other image contents of interest to determine measurements in the 
scene. 

According to another embodiment, however, measurements may be accurately made 
in a scene having significantly larger dimensions than a reference target placed in the scene. 
In particular, according to one embodiment, a series of similar images of a scene that are 
obtained from a single camera location may be processed in a "scale-up" procedure, wherein 
the images have consecutively larger scales (i.e. the images contain consecutively larger 
portions of the scene). In one aspect of this embodiment, camera calibration information is 
interpolated from the smaller-scale images to the larger-scale images rather than extrapolated 
throughout a single image, so that relatively smaller reference objects (e.g., a reference target) 
placed in the scene may be used to make accurate measurements throughout scenes having 
significantly larger dimensions than the reference objects. 
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In one example of this implementation, the determination of camera calibration 
information using a reference target is essentially "bootstrapped" from images of smaller 
portions of the scene to images of larger portions of the scene, wherein the images include a 
common reference plane. For purposes of illustrating this example, with reference to the 
illustration of a scene including a cathedral as shown in Fig. 26, three images are considered; 
a first image 600 including a first portion of the cathedral, a second image 602 including a 
second portion of the cathedral, wherein the second portion is larger than the first portion and 
includes the first portion, and a third image 604 including a third portion of the cathedral, 
wherein the third portion is larger than the second portion and includes the second portion. In 
one aspect, a reference target 606 is disposed in the first portion of the scene against a front 
wall of the cathedral which serves as a reference plane. The reference target 606 covers an 
area that is approximately equal to or greater than one-tenth the area of the first portion of the 
scene. In one aspect, each of the first, second, and third images is obtained by a camera 
disposed at a single location (e.g., on a tripod), by using zoom or lens changes to capture the 

different portions of the scene. 

In this example, at least the exterior orientation of the camera (and optionally other 
camera calibration information) is estimated for the first image 600 based on reference 
information associated with the reference target 606. Subsequently, a first set of at least three 
widely spaced control points 608A, 608B, and 608C not included in the area of the reference 
target is identified in the first image 600. The relative position in the scene (i.e., coordinates 
in the reference coordinate system) of these control points is determined based on the first 
estimate of exterior orientation from the first image (e.g., according to Eq. (11)). This first 
set of control points is subsequently identified in the second image 602, and the previously 
determined position in the scene of each of these control points serves as the reference 
information for a second estimation of the exterior orientation from the second image. 

Next, a second set of at least three widely spaced control points 610A, 610B, and 
610C is selected in the second image, covering an area of the second image greater than that 
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covered by the first set of control points. The relative position in the scene of each control 
point of this second set of control points is determined based on the second estimate of 
exterior orientation from the second image. This second set of control points is subsequently 
identified in the third image 604, and the previously determined position in the scene of each 
of these control points serves as the reference information for a third estimation of the exterior 
orientation from the third image. This bootstrapping process may be repeated for any number 
of images, until an exterior orientation is obtained for an image covering the extent of the 
scene in which measurements are desired. According to yet another aspect of this 
embodiment, a number of stand-alone robust fiducial marks may be placed throughout the 
scene, in addition to the reference target, to serve as automatically detectable first and second 
sets of control points to facilitate an automated scale-up measurement as described above. 

13. Automatic Intersection or Bundle Adjustments using Multiple Images 
According to another embodiment of the invention involving multiple images of the 
same scene obtained at respectively different camera locations, camera calibration information 
may be determined automatically for each camera location and measurements may be 
automatically made using points of interest in the scene that appear in each of the images. 
This procedure is based in part on geometric and mathematical theory related to some 
conventional multi-image photogrammetry approaches, such as intersection (as discussed 
above in Section G of the Description of the Related Art) and bundle adjustments (as 
discussed above in Section H of the Description of the Related Art). 

According to the present invention, conventional intersection and bundle adjustment 
techniques are improved upon in at least one respect by facilitating automation and thereby 
reducing potential errors typically caused by human "blunders," as discussed above in Section 
H of the Description of the Related Art. For example, in one aspect of this embodiment, a 
number of individually (i.e., uniquely) identifiable robust fiducial marks (RFIDs) are disposed 
on a reference target that is placed in the scene and which appears in each of the multiple 
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images obtained at different camera locations. Some examples of uniquely identifiable 
physical attributes of fiducial marks are discussed above in Section G3 of the Detailed 
Description. In particular, a mark similar to that shown in Fig. 16A may be uniquely formed 
such that one of the wedged-shaped regions of the mark has a detectably extended radius 
compared to other regions of the mark. Alternatively, a fiducial mark similar to that shown in 
Fig. 16A may be uniquely formed such that at least a portion of one of the wedged-shaped 
regions of the mark is differently colored than other regions of the mark. In this aspect, 
corresponding images of each unique fiducial mark of the target are automatically referenced 
to one another in the multiple images to facilitate the "referencing" process discussed above in 
Section H of the Description of the Related Art. By automating this referencing process using 
automatically detectable unique robust fiducial marks, errors due to user blunders may be 

virtually eliminated. 

In another aspect of this embodiment, a number of individually (i.e., uniquely) 
identifiable stand-alone fiducial marks (e.g., RFIDs that have respective unique identifying 
attributes and that are printed, for example, on self-adhesive substrates) are disposed 
throughout a scene (e.g., affixed to various objects of interest and/or widely spaced 
throughout the scene), in a single plane or throughout three-dimensions of the scene, in a 
manner such that each of the marks appears in each of the images. As above, corresponding 
images of each uniquely identifiable stand-alone fiducial mark are automatically referenced to 
one another in the multiple images to facilitate the "referencing" process for purposes of a 
bundle adjustment. 

It should be appreciated from the foregoing that either one or more reference targets 
and/or a number of stand-alone fiducial marks may be used alone or in combination with each 
other to facilitate automation of a multi-image intersection or bundle adjustment process. The 
total number of fiducial marks employed in such a process (i.e., including fiducial marks 
located on one or more reference targets as well as stand-alone marks) may be selected based 
on the constraint relationships given by Eqs. (15) or (16), depending on the number of 



-131 - 



parameters that are being solved for in the bundle adjustment. Additionally, according to one 
aspect of this embodiment, if the fiducial marks are all located in the scene to lie in a 
reference plane for the scene, the constraint relationship given by Eq. (16), for example, may 
be modified as 



2 jn >l Cj + In , 



(19) 



where C indicates the total number of initially assumed unknown camera calibration 
information parameters for each camera, n is the number of fiducial marks lying in the 
reference plane, and/ is the number of different images. In Eq. (19), the number n of fiducial 
marks is multiplied by two instead of by three (as in Eqs. (15) and (16) ), because it is 
assumed that the z-coordinate for each fiducial mark lying in the reference plane is by 
definition zero, and hence known. 

14. Site Surveys using Automatically Linked Multiple Images 
According to another embodiment, multiple different images containing at least some 
common features may be automatically linked together to form a "site survey" and processed 
to facilitate measurements throughout a scene or site that is too large and/or complex to obtain 
with a single image. In various aspects of this embodiment, the common features shared 
between consecutive pairs of images of such a survey may be established by a common 
reference target and/or by one or more stand-alone robust fiducial marks that appear in the 
images to facilitate automatic linking of the images. 

For example, in one aspect of this embodiment, two or more reference targets are 
located in a scene, and at least one of the reference targets appears in two or more different 
images (i.e., of different portions of the scene). In particular, one may imagine a site survey 
of a number of rooms of a built space, in which two uniquely identifiable reference targets are 
used in a sequence of images covering all of the rooms (e.g., right-hand wall-following). 



132 



Specifically, in this example, for each successive image, only one of the two reference targets 
is moved to establish a reference plane for that image (this target is essentially "leapfrogged" 
around the site from image to image), while the other of the two reference targets remains 
stationary for a pair of successive images to establish automatically identifiable link points 
between two consecutive images. At corners, an image could be obtained with a reference 
target on each wall. At least one uniquely identifying physical attribute of each of the 
reference targets may be provided, for example, by a uniquely identifiable fiducial mark on 
the target, some examples of which are discussed above in Sections 13 and G3 of the Detailed 
Description. 

According to another embodiment, at least one reference target is moved throughout 
the scene or site as different images are obtained so as to provide for camera calibration from 
each image, and one or more stand-alone robust fiducial marks are used to link consecutive 
images by establishing link points between images. As discussed above in Section G3 of the 
Detailed Description, such stand-alone fiducial marks may be provided as uniquely 
identifiable marks each printed on a self-adhesive substrate; hence, such marks may be easily 
and conveniently placed throughout a site to establish automatically detectable link points 

between consecutive images. 

In yet another embodiment related to the site survey embodiment discussed above, a 
virtual reality model of a built space may be developed. In this embodiment, a walk-through 
recording is made of a built space (e.g., a home or a commercial / industrial space) using a 
digital video camera. The walk-through recording is performed using a particular pattern 
(e.g., right-hand wall-following) through the space. In one aspect of this embodiment, the 
recorded digital video images are processed by either the image metrology processor 36 of 
Fig. 6 or the image metrology server 36A of Fig. 7 to develop a dimensioned model of the 
space, from which a computer-assisted drawing (CAD) model database may be constructed. 
From the CAD database and the image data, a virtual reality model of the space may be made, 
through which users may "walk through" using a personal computer to take a tour of the 
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space. In the network-based system of Fig. 7, users may walk through the virtual reality 
model of the space from any client workstation coupled to the wide-area network. 
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J: Orientation Dependent Radiation Analysis 
Jl. Introduction 

Fourier analysis provides insight into the observed radiation pattern emanated by an 
exemplary orientation dependent radiation source (ODR), as discussed in section G2 of 
the detailed description. The two square-wave patterns of the respective front and back 
gratings of the exemplary ODR shown in Fig 13A are multiplied in the spatial domain; 
accordingly, the Fourier transform of the product is given by the convolution of the 
transforms of each square-wave grating. The Fourier analysis that follows is based on the 
far-field approximation, which corresponds to viewing the ODR along parallel rays, as 
indicated in Fig 12B. 

Fourier transforms of the front and back gratings are shown in Figs 27, 28, 29 and 
30. In particular, Fig 27 shows the transform of the front grating from -4000 to +4000 
[cycles/meter], while Fig 29 shows an expended view of the same transform from -1500 to 
+ 1500 [cycles/meter]. Similarly, Fig 28 shows the transform of the back grating from - 
4000 to +4000 [cycles/meter], while Fig 30 shows an expanded view of the same transform 
from -1575 to + 1575 [cycles/meter]. For the square wave grating, power appears at the 
odd harmonics. For the front grating the Fourier coefficients are given by: 



• (_!)(*-!)/» II A; odd 
F(kfj) = \ 

0 otherwise 

And for the back grating the Fourier coefficients are given by: 



(20) 



(21) 

0 otherwise 

£ 

where: 

/, is the spatial frequency of the front grating [cycles/meter]; 
f b is the spatial frequency of the back grating [cycles/meter]; 
F (/) is the complex Fourier coefficient at frequency / ; 
k is the harmonic number, / = k ff or / = k f b ; 

Ax b [meters] is the total shift of the back grating relative to the front grating, 
defined in Eqn (26) below. 

The Fourier transform coefficients for the front grating are listed in Table 1. The 
coefficients shown correspond to a front grating centered at x = 0 (i.e., as shown in 
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Fig 13A). For a back grating shifted with respect to the front grating by a distance Ax b , 
the Fourier coefficients are phase shifted by f**»f*«\ as seen in Eqn (21). 



f = kff [cycles/meter] 


Jfc 


F(kf f ) [Amplitude] 








-5// = -2500 
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(-i) 3 H = -°- 064 
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C" 1 )^! = "°- 318 


0// = 0 
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Iff = 500 


1 


("1)^1 = "0.318 
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(-i) 2 H = 0106 
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Table 1: Fourier transform coefficients for the ODR front grating square-wave pattern; 
f f = 500 [cycles/meter] is the spatial frequency of the front grating. 

Convolution of the Fourier transforms of the ODR front and back gratings corresponds 
to multiplication of the gratings and gives the Fourier transform of the emanated 
orientation-dependent radiation, as shown in Figs 31 and 32. In particular, the graph of 
Fig 32 shows a closeup of the low-frequency region of the Fourier transform of orientation- 
dependent radiation shown in Fig 31. 

Identifying the respective coefficients of the front and back grating Fourier transforms 

as: 



Front: 



Back: 



a_3, o_i, ao, Oi, 03, • ■ 



then, for the case of f b > // , the coefficients of the Fourier transform shown in Fig 
32 (i.e., the center-most peaks) of the orientation-dependent radiation emanated by the 
ODR are given in Table 2, where: 

F = min(//, f b ) is the smaller of the grating spatial frequencies; 
Frequencies lying in range between -F to +F are considered; 
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&f _ f f -f b , is the frequency difference between the front and back gratings, 
(A/ can be positive or negative). 



/ 


Coefficient 






-3A/ 




-1A/ 




0 


<*o ao = (l) 


1A/ 




3A/ 









Table 2: Coefficients of the central peaks in the Fourier transform of the orientation- 
dependent radiation emanated by an ODR {fa > //)• 

These peaks correspond essentially to a triangular waveform having a frequency 
/jvf = |A/| and a phase shift of 



i/ = 360Ax 6 / 4 [degrees] 



(22) 



where v is the phase shift of the triangle waveform at the reference point x = 0. An 
example of such a triangle waveform is shown in Fig 13D. 

With respect to the graph of Fig 31, the group of terms at the spatial frequency 
of the gratings (i.e., approximately 500 [cycles/meter]) corresponds to the fundamental 
frequencies convolved with the DC components. These coefficients are given in Table 

3. The next group of terms correspond to sum frequencies. They are given in Table 

4. Groups similar to that at (/, + h) occur at intervals of increasing frequency and in 
increasingly complex patterns. 



/ 


Coefficient 


// 


_ „ _ in 


ft 


aiao = e x±* b u b 2«nLi 


-ff 


„ „ _ 1 1 1 
a 0 a-i - 2^1 


-h 


«-l«0 — e 2 7r 1 



Table 3: Fourier coefficients at the fundamental frequencies (500 and 525 [cycles/meter]). 
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f 




ff + h 




(// + /»)-2A/ 




(// + /») + 2A/ 




(/ / + / t )-4A/ 






— 



Table 4: Fourier coefficients at the sum frequencies. 

As discussed above, the inverse Fourier transform of the central group of Fourier terms 
shown in Fig 31 (i.e., the terms of Table 2, taken for the entire spectrum) exactly gives a 
triangle wave having a frequency f M = \Af\, phase shifted by v = 360 Az 6 / 6 [degrees]. 
As shown in Fig 13D, such a triangle wave is evident in the low-pass filtered waveform of 
orientation-dependent radiation. The waveform illustrated in Fig 13D is not an ideal a 
triangle waveform, however, because: a) the filtering leaves the 500 and 525 [cycle/meter] 
components shown in Fig 31 attenuated but none-the-less present, and b) high frequency 
components of the triangle wave are attenuated. 

Fig 33 shows yet another example of a triangular waveform that is obtained from an 
ODR similar to that discussed in Section G2, viewed at an oblique viewing angle (i.e., 
a rotation) of approximately 5 degrees off-normal, and using low-pass filtering with a 
3dB cutoff frequency of approximately 400 [cycles/meter]. The phase shift 408 of Fig 33 
due to the 5° rotation is -72", which may be expressed as a lateral position, x T , of the 
triangle wave peak relative to the reference point x = 0: 



Xt = "Z 360 [meters] 
Jm 



(23) 



where x T is the lateral position of the triangle wave peak relative to the reference point 
x = 0 and takes a value of -0.008 [meters] when f M = 25 [cycles/meter] in this example. 

The coefficients of the central peaks of the Fourier transform of the orientation- 
dependent radiation emanated by the ODR (Table 2) were derived above for the case 
of a back grating frequency greater than the front grating frequency (/ 6 > //)- When 
the back grating frequency is lower than that of the front, the combinations of Fourier 
terms which produce the low-frequency contribution are reversed, and the direction of 
the phase shift of the low-frequency triangle waveform is reversed (i.e., instead of moving 
to the left as shown in Fig 33, the waveform moves to the right for the same direction of 
rotation. This effect is seen in Table 5; with (// > ft), the indices of the coefficients are 
reversed, as are the signs of the complex exponentials and, hence, the phase shifts. 
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-1A/ 
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1A/ 


a_i ai = e n ^ l2 


3A/ 


^-j(Aa;b3A27r) 1 1 







Table 5: Coefficients of the central peaks in the Fourier transform of the orientation- 
dependent radiation emanated from an ODR (// > h)- 

J2. 2-D Analysis of Back Grating Shift with Rotation 
From the point of view of an observer, the back grating of the ODR (shown at 144 in Fig 

mTshiftLla^ 
bU^). The two dimensional (2-D) case is considered in this subsecfcon because * 
M*. the properties of the ODR and because it is the applicable analyse when an 
ODR Ranged to measure rotation about a single axis. The process of back-gratmg 
shift is illustrated in Fig 12A and discussed in Section G2. 

J 2.1. The far-field case, with refraction 
In the ODR embodiment of Fig 11, the ODR h M primary axis 130 and secondary axis 
,32 The X and Y axes of the ODR coordinate frame are denned such that - vector 
X D ^parallel to primary axis 130, and unit vector ry B s # is paraUel to the 
solidary axis 132 (the ODR coordinate frame is further described in Section L2.4) 
StL 'X D € * indites that >X D is a vector of three 

numbers, for example 'X B = [ 1 0 0 f. This notation w,H be used £ md»» th 
si.es of vectors and matrices below. A special case is a real scalar whrch is m R , 

6 T d^cfbelow in connection with Pig 11, K C * * - shift of the 

bac, grating due to rotation. I. the genera, three-dimensional (3-D ^.dered ,n 
section 33 below, and for the ODR embodiment described m connection with Fig 11, the 
rrsnKt . of tne observed radiation pattern is determined in part by the component 
of {>* which is parallel to the primary axis, said component bemg given by: 
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where 5 Db x [meters] is the component of 8»x which contributes to determination of 
phase shift v. In the special, two-dimensional (2-D) case described in this section we 
are always free to choose the reference coordinate frame such that the X axis of the 
reference coordinate frame is parallel to the primary axis of the ODR, with the result 
that r Xl = f 1 0 0 ] T and 5 Db x = 5 b x (1) 

A detailed view of the ODR at approximately a 45" angle is seen in Fig 34. The 
apparent shift in the back grating relative .to the front grating due to an obhque vaew 
angle, 5 Db x, (e.g., as discussed in connection with Fig 12B) is given by: 

8 Db x = zit^& [meters] ( 25 ) 
The angle of propagation through the substrate, 0 1 , is given by Snell's law: 

m sin 9 = n 2 sin 0' 



or 

d = sin" 1 f 

Where 



0' = sin- 1 (^-sinfl) 



6 is the rotation angle 136 (e.g., as seen in Fig 12A) of the ODR [degrees], 
& is the angle of propagation in the substrate 146 [degrees], 
zi is the thickness 147 of the substrate 146 [meters], 

ni , n 2 are the indices of refraction of air" and of the substrate 146, respectively. 
The total primary-axis shift, Ax t , of the back grating relative to the front grating is the 
sum of the shift due to the rotation angle and a fabrication offset of the two gratmgs: 

Arr 6 = 5 m x + x 0 = z« tan (sin" 1 sin fl)) + x» ( 2( 5) 

Where 

Ax b G R 1 is the total shift of the back grating [meters], 

x 0 G R 1 is the fabrication offset of the two gratings [meters] (part of the 

reference information). 

Accordingly, for x 0 = 0 and 9 = 0°, i.e., normal viewing, from Eqn (26) it can be seen 

that Aa; 6 = 0 (and, hence, v = 0 from Eqn (22)). 
Writing the derivative of Eqn (26) w.r.t. 9 gives: 

d5 Db x _ ni cos (9) 
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Writing the Taylor series expansion of the 5 Db x term of Eqn (26) gives: 




f n 2 v n l I 120 + 4 12ti2 , 

5 Db x _ m ^ "i H + 5fe) pa . _A_ ^ + o (27) 

zj n 2 "2 2 

Using the exemplary indices of refraction n, = 1.0 and n 2 = 1.5, the Taylor series 
expansion becomes 

— = 318O 0+ 27V18OJ 1620 Vl80j ^ J 

= 0.666667 ^ + 0.037037 - 0.0191358 (^j)* 0* + 0 (* 7 ) 

where 0 is in [degrees]. . 

One sees from Eqn (28) that the cubic and quintic contributes to 6 x are not 
necessarily insignificant. The first three terms of Eqn (28) are plotted as a I unc W 
angle in Fig 35. From Fig 35 it can be seen that the cubic term makes a part per thousand 
contribution to 6 "x at 10" and a 1% contribution at 25° 

Accordingly, in the far-field case, v (or x T ) is observed from the ODR (see Fig 

According , . ig evaluate d 

33), divided by /„ to obtain Ax 6 (from Eqn (22)), and nnaliy rxra k , 

to determine the ODR rotation angle 9 (the angle 136 in Fig 34). 

J2.2. The near-field case, with refraction 
ODR observation geometry in the near-field is illustrated in Fig 36. Where* > in Fig 12B 
all rays are shown parallel (corresponding to the camera located far from the ODR) in 
Fig 36, observation rays A and B are shown diverging by angle 

From Fig 36, it may be observed that the observation angle * is given by: 

_! / ^x(l) cos0 N (29) 

where fx € R> [meters] is the observed location on the observation (front) surface 128A 
If le ODR; 'x(l) 6 * l-ters] is the X-axis component of *; Ml ) = 0 
to the intersection of the camera bearing vector 78 and the reference point 125A (x _ 0) 
on the observation surface of the ODR; the camera bearing vector 78 extends from the 
reference point 125A of the ODR to the origin 66 of the camera coordi -te system; , „ 
is the length 410 of the camera bearing vector, (i.e., the distance between the ODR and 
the camera origin 66); and 9 is the angle between the ODR normal vector and the camera 
bearing vector [degrees]. 
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The model of Fig 36 and Eqn (29) assumes that the optical axis of the camera 
intersects the center of the ODR region. From Fig 36 it may be seen that in two 
dimensions the angle between the observation ray B and an observation surface normal 
at *x0) is 9 + accordingly, from Eqn (25) and Snell's law (see Fig 34, for example) 



5 Db x = zi tan (sin" 1 sin (9 + 1/>))) 



(30) 



Because V varies across the surface, 5 Db x is no longer constant, as it is for the far-field 
case. The rate of change of 6 Db x along the primary axis of the ODR is given by: 

= dJ^W tan (sin- sin (0 + *))) -A* (31) 

The pieces of Eqn (31) are given by: 

d 5 Db x ni cos (9 + V>) (32) 

d * ~" n2 (l-fe S in(f + ^)) 2 ) 1 

And 1 

dlf) _ _ Zcom COS 9 (33) 

Thffl ~ 4m. + 2 z c „m sin0 fx(l) + *x{l) 2 
The term is significant because it changes the apparent frequency of the back 

grating. The apparent back-grating frequency, fi is given by: 



fb = fb JTxO)~ fb \ dfx(l)) 



cycles 



meter 



(34) 



From Eqns (31) and (33) it should be appreciated that the change in the apparent 
frequency f' b of the back grating is related to the distance Zcam . The near-field effect 
causes the swept-out length of the baek grating to be greater than the swept-out length 
of the front grating, and so the apparent frequency of the back grating is always increased. 
This has several consequences: 

. An ODR comprising two gratings and a substrate can be reversed (rotated 180" 
about its secondary axis), so that the back grating becomes the front and vice versa. 
In the near-field case, the spatial periods are not the same for the Moire patterns 
seen from the two sides. When the near-field effect is considered, f' M € R 1 , the 
a pparent spatial frequency of the ODR triangle waveform (e.g. as seen at 126A in 
^Equations (32) and (33) have the intriguing property of canceling curvature when m = » 2 - This 
numerical result has not yet been established algebraically. 



(35) 
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Fig 33) will depend on the apparent back-grating frequency ft 

[ cycles ] 

I'm = \ff ~ f>>\ [^ete^J 

When sign (/, - /„) = sign (// - ft) we may right: 

, , d8 Db x . (f , >, 

where the sign(.) function is introduced by bringing the differential terra out from 
t absolute^lue. If the back grating has the lower spatial frequency, t e effete 
increase in h due to the near-field effect reduces /, - ft, and f M is reduced. 

This effect permits differential mode sensing of z cam . 

In contrast, when the ODR and camera are widely separated and the far-field 

approximation is valid, the spatial frequency of the Moire pattern (i.e., the tnang 

waveform of orientation-dependent radiation) is given simply by /„ - \ff h\ 
waveiorm oi Thus in the far-field case, the spatial 

and is independent of the sign of (// - h)- Inus, in tne 

frequency (and similarly, the period 154 shown in Figs 33 and 13D) of the ODR 
~ted radiation is independent of whether the higher or lower frequency 

grating is in front. 

. There is a configuration in which the Moire pattern disappears in the near-field 
case: for example, given a particular combination of ODR parameters /, and f b , 
and pose parameters 9 and z cam in Eqn (31): 



f'M = \ff-f'»\ = 



ff-h- fi 




. Front and back gratings with identical spatial frequencies, /, = />, produc a Moir 
pattern when viewed in the near field. The near-field spatial frequency f M of the 
Ml pattern (as given by Eqn (35)) indicates the distance Zcam to the camera if 
the rotation angle 9 is known (based on Eqns (31) and (33)). 

J2.S. Summary 

Several useful engineering equations can be deduced from the foregoing. 

. Detected phase ang!e , is given in terms of ( ». (assuming the fabrication offset 
x 0 = 0, from Eqns (22) and (4)): 

v = 6 Db xf b 3S0 [degrees] 
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• 5 Db x as a function of , A*™ e '• 

The position x T of a peak [e.g., p omnTia tPd bv an ODR, relative to 

waveform of the orientation-dependent radiation elated by an 
the reference point 125A (x = 0). Taking the fabrication offset x 0 
XT of the triangular waveform is given by 

XT = i-„ = JL/.z.tan (sin" 1 + * |*W-r)) * £ ^ W 

w here . is in degrees, and wherein the first term of the Taylor series expansion in 
Eqn (27) is used for the approximation in Eqn (3b). 

*om Eqn 06) an ODR sensitivity may be defined as W - ^ and may be 
approximated by: ^ ^ ; , (37) 

. At hreshold angle ^ degrees for the ^^^Z^^ 
less than a 1% effect (i.e., the approximation in Eqn (36) has 
1%) is given by: ^ , ^ (w) 

(From the cubic term of the Taylor series expansion, Eqn (27)). Using * - 
1.0, and n 2 = 1.5 gives: ^ = ^ 

. Threshold for the length of the camera bearing vector, ^. the near-field effect 
to give a change in f' M of less than 1%: 

t d6 Db x^. %f , (39) 

Evaluating Eqn (35) with m = 1.0, n, = 1.5 and 9 = 0" gives 



d^x 
d 'x(l) 



i 0.65 



zi 

'''cam 
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and substituting into Eqn (39) gives: 



0.65 /t r (40) 

Z\ <^ ^cam 



0.01% I'm 

Accordingly, E,n (40) provides one criterion for 

field observation given particular par— « * * 

may be denned as a design cntenon for the ODR 122A based 

application as , (41) 

FOM = 1 r—, V 

v, FOM > 0 01 generally indicates a reliably detectable near-field efface, 

where an FOM > U.U1 ge / accurately measurable distance w 

n r,H an FOM > 0.1 generally indicates an accurately 

and an * u-Jw ^ & otherwise, the intensity of the 

^"ft, - ^ chosen to be very small, thereby leasing — * 

to Zcam' 

an ODR similar to that described above in connection with various figures may 

the ODR. based on -^^'^S^*^^ - ~~ 

— f , ff t t.Sh « 0 of the camera bearing vector 78) ma, be 
ODR and the camera ongi Irving the spatial frequency ft, (or the period 
determined based on the angle 0 and observing . ODR frcim 

154 shown in figs 33 and 13D) of the Moire pattern produced by the ODR, from 

(31), (33), and (35). 

JS. Genera! 3-D Analysis ol Back Gratis Snift in tte Near Fieli «tt Rotation 

. vft „f .he back Mating as seen from the camera position determines 
The apparent shift of the back g™">S determined in three 

the phase shift of the Moire pattern. ™s parent **- ^ ^ ^ 

dimensions by vector analysis of the line of sight. Key terms 

^e* is the vector 412 from the camera origin 66 ,0 a point /. of the front (i.e., 

observation) surface 128 of the ODR 122A; 
V 2 e R 3 is the continuation of the vector V, through the ODR substrate 146 to the back 

surface (V4 is in general not collinear with V, because of refraction), 
/* 6 if> is the point where vector VI strikes the front surface (the coordinate frame of 
111- is indicated by the left superscript, coordinate fr^ies are dossed 
further in Section L2.4); 
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•« e R* is the point where vector V 2 strikes the back surface. 

JS.l. Determination of phase shift v as a function of fx, v (/*) 
In n dimensions, Snell's law may be written: 

(42) 

nsV^nxVfS (43) 

the surface normal. Using fcqn ^ x f v 2 can be computed 

as a unit vector (e.g., in reference coordinates) V" - [ 0 0 1] , 

by: 

yj. = [y l( i : 2)/|V 1 |, Of 5 V 2 - L = ^V'i L 
V 2 =[V 2 M1:2), -^H^f^)] 

Using 5"* ('*), * ('«) - the Moire pattern phase, is given by: 

6 D "x= r Xl5 b x 

.here ^ is the location of the origin of ^J^ZTJ^ZZ 

coordinates; 5 Db x e R 1 [meters] is the component of <5 * € K 
ODR primary axis and which determines v. 

u ^x) = v, + 360 (f b - ff) D 's + 360 fb & Db x [deg] 

where 

„ (/*) € i? 1 is the phase of the Moire pattern at position >s € R 3 ; 
^/xGi? 1 , Df x='Xl fx; 

'X D Gtfisa unit vector parallel to the primary axis of the ODR. 



(45) 



(46) 



(47) 



- 146 - 

The model of finance used for camera calibration - given by the W harmonic of the 
triangle waveform: + W 

- . nr»R retrion and oi is the amplitude of the 
where a„ is the average luminance across the ODR region, 

Luminance variation. nnR recion: i/ 0 , 

nation, (*7> and (48, ^ ^^^^T^S 1» the 

j „ porameter v« is a property of the UUtt region, ^ 
a 0 and oi. Parameter i/ 0 u» f f j „ nTnpra . aperture, shutter speed, 

ODR was — JtT^^ once a, P art of a 

„, H e estimated each time the orientation of the ODR . elated. 
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K: Landmark Detection Methods 

Three methods are discussed below for detecting the presence (or absence) of a 
JT-^ — phase rotation ^^^^ 
ed gM anai y sis. The Z of the methods 

IT^Z^Z the methods m W be empio.ed in various 
combinations to refine the detection process. 

Kl Cumulative Phase rotation analysis 

seen at 300 m Fig 19. ine rami fi i tPrmtr in Fig 22A. This scan 

■ . , a n AV5imn ie luminance curve is seen before filtering in "8 
scanned signal. An example iumm.ui no 

^region, which in this — - - ^ ™ ^ _ding . 

The raw scanned signal of Fig. 22A is nite 
one embodiment, with a two-pa. hnear, ^ - 
seen as the luminance curve of Fig 22B. Other examp 

shown in Figs 16B, 17B and 18B. ^tsmtaneous phase rotation of a 

*- e ' te -tr ^^"Irt-timeKonrier 
t^Z"^^ * estimating phase an gl e at each samp, This 

latter method comprises: 

and adding the segment from 0° to 10" after the end. 
2. Constructing the quadrature signal according to: 



Where 

a (i) € CMs a complex number (indicated by «« 6 C 1 ) representing 
the phase of the signal at point (i.e., pixel sample) t; 
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A (i) € R 1 is the filtered luminance at pixel i (e.g., t is an index on the 
pixels indicated, such as at 328, in Fig 20); 

A € Z+ is a positive integer (indicated by A € Z+) offset, given by: 

/ 360» .360^ (50 ) 
A = round l47iV / N a ) 
N, is the number of points in the scanned path, and N is the number of 
separately identifiable regions of the mark; 
j is the complex number. 
3. The phase rotation 6* <= B> [degrees] between sample i - 1 and sample i is given 
by: 

5 7fc = atan2(im(6(i)),re(b(i))) ( 51 ) 

where 



and where a*n2(,0 is the 2-argument arctangent function as provided, for 
example, in the C programming language math library. 
4. And the cumulative phase rotation at scan index i, i» € R\ is given by: 

r (52) 

Examples of cumulative phase rotation plots are seen in Figs 160, 17C, 180 and 22C_ 
L Z l ar Figs 16C, HO and 18C show cumulative phase rotation plots when a mark 
£ pr^nt wherL Fig 22C show, a cumulative phase rotation plot when no mart - 
T, !k of these f.™ * is plotted against * € B\ where * is the scan angle 
rtZ Z ^1T« U-i «. shown at 3« in Fig 20. The robust nducial mar. 
^D) Iwn at 320 in Fig 19 would give a cumulative phase rotation curve M 
£pe of JV when plotted against *. In other words, for normal vewng angle and when 
the scanning curve is centered on the center of the MID 

m = N4>, 

In each of Figs 160, 170, 18C and 220 the „ curve is shown at 366 and the N * curve 
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to shown at 349. Compared with Figs 16C, 170 and 18C, the deviation in Fig 22C from 
offteT curve, 366, from the N * reference line 349 is very large. Th,s devrafon » the 
hi for the cumulative phase rotation analysis. A performance measure for detect 

is- rms([A]) (53) 

•J l = _ tin 



Where 



rms ([A]) is the RMS value of the (possibly filtered) luminance signal [A] , and 
*([„]) is the RMS deviation between the N <f> reference line 349 and the 
cumulative phase rotation of the luminance curve: 



e([ri[) = TtaB([v]-N[4>]) ; 



(54) 



and where [A], [ V ], [<] indicates vectors of the corresponding variables 

over the N s samples along the scan path. 
The offset 362 shown in Fig ISA indicates the position of the center of the mark with 
"to the center of the scanning path. The offset and tilt of the mark are found by 
fitlg 7o first and second harmonic terms the difference between the cumulate phase 
rotation, e.g. 346, 348, 350 or 366, reference line 349: 



cos 



([$) sin ([<£]) cos (2[^) sin(2[$) 



(55) 



Where 



Eqn (55) implements a least-squared error estimate of the cosine and sine 
pL of the first and second harmonic contributions to the cumulate phase 
curve; 

and W is the vector of sampling angles of the scan around the closed path 
(i.e., the X-axis of Figs 16B, MO. 1TB. 170, 18B, 18C , 22B and 220). 



This gives: 



„ (*) = N * + n c (1) cos (*) + n c (3) sin (*) + n c (3) cos (2*) + H c (4) sin (2*) (56) 
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wh ere the vector H c 6 copses coefficients of cosine and sine parser the first and 
Ilnd harmonic; these are converted to magnitude and phase by wntmg. 

(57) 



r, (0) = N <t> + A lC os + ft) + M<x» (2<£ + ft) 



Where 



Al = /n c (i) 2 + n c W 

ft = -atan2 (n c (2) , n c (1)) [degrees] 

A 2 = > /lU3) 2 + n c (4f 

ft = -atan2 (II C (4) , II C (3)) [degrees] 



Offset and tilt of the fiducial mar, make contributions to the first and second harmonics 



frafrrt. First Harmonic Second Harmonic 


Offset 


X 


X 


Tilt 







So offset and tilt can be determined by: 

!. Determining the offset from the measured first harmonic; 

2. Subtracting the influence of the offset from the measured second harmonic; 

3. Determining the tilt from the adjusted measured second harmonic. 
1. The offset is determined from the measured first harmonic by: 



sin (ft) 
cos (ft) 



[pixels] (58) 



2 . The contribution of offset to the cumulative phase rotation is given by: 

■no (0 = ^ cos W + ft) + A2fl cos { - 2<i> + ^ 

Where r, 0 is the contribution to r, due to offset, and with 

1 YAxN 2 2tt 



A -I/dlV— ; ft a = 90 + 2ft 
Ma ~ 2\NJ 360 ' P2a 
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Subtracting the influence of the offset from 
adjusted measured second harmonic: 

n' c (3) = n c (3) - A 2a cos {fa) 

II' C (4) = II C (4) - >l2aSin (#2a) 



the measured second harmonic gives the 



3. And finally, 



A 26 = VnU3) 2 +n' c (4) 2 

0 26 = -atan2(IT c (4),n' c (3)) 

Where the second harmonic contribution due to tilt is given by: 

V2b(<t>) = A2 b cos(2<f) + M 



(59) 



The tilt is then given by: 



r t = l-2A 26 ^ [rad] 



(60) 



pt= te^r [deg] 

w here Pt is the rotation to the tilt axis, and * - cos- (n) is the tilt angle. 

Kl.l. Quadrature Color Method 

■ ■ filial mark can contain additional information that can be 
With color imaging a fiducial mark can c alco _ ithm A quadrature color 

exploited to enhance the robustness of the detection algo^hm. q 
RPID is described here. Using two colors to ^^^TZ synthesis 
possible to directly generate phase rotation on the * ^ reduced 

{n\ The results -obtained at the cost of using a color camera 

An example . shown m F« 23A- The ^ ^ with 

yellow, in a rotating pattern of black-blue-green yellow 

the combination of bine and yellow. obtained; 
If the color image is filtered to show only blue light, the image ot r ig 
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, i nf Fi* 23A lie at four corners of a square centered on the average luminance 
the four colors of Fig 23A he at to embodim ent, the color intonates 

rrrrr^rx- ~ m - — 

directly determined by: 

(61) 



a(i)=(Ay(i)-\) + ^( At(i)_ ^ 



algorithm, with the advantages of: 

described with Eqn (49) above; 

presence of all four colors along a scanning path. 

such as shown in Fig 40. For very high robustness, 
applied to both the blue and yellow filtered images. 

K2 Regions analysis 
of arbitrary regions in an image are evaluated For example . 

of an image containing a mark can be threshold, p „onc^a b ^ 
with distinct connected regions, as Been m F.g 39. The bmary 

regions of contiguous black pixels. regions. The various 

Contiguous grcups of black pixels may be ^°^T^Zenc* quantities, 
properties of the labeled regions can then be "d- ed, and for each 
For example, 165 distinct black regions in the ^J^lZ TL.^ o, which is 
region a report is generated based on the measured W*-' J ^ 
Jn in Table 6. In short, numerical quanft.es are computed for each 
for each contiguous region. 
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sample of properties of distinct black regions in Fig 39. 



Table 6: Representative 

- - i a ™th it is possible to identify each labeled region touched by the 

Jsr r : vs. . - - - ■ - * — 

identifiable regions proceeds by: 

1. Establishing the scan pixels encircling a center; 

2. Determining the labeled regions touched by the scan pixels; 

3. Throwing out any labeled regions with an area less than a minimum threshold 
number of pixels; 

4. If there are not JV regions, reject the candidate; 

5. If there are JV regions, compute a performance measure according to: 

C = 



~~ N 



vo. 



V Ci = Ci-C 



i = atan2(y Ci (2), V Ci (l)) 



J 2 = 



i/Ef /2 {(^- Ai * )2/(( ^ + ^* )/2) 
+ (M i -M i .) 2 /((Mi + M i .)/2)' , 

+ (m i -m i .) 2 /((" l i + m **)/ 2 ) 
+ (Pi - &i>? / ((^i + M 
+ (^-*«OV((*« + **0/2) 2 } 



(62) 
(63) 
(64) 
(65) 
(66) 



Where 



d is the centroid of the i th region, % 



€l---JV; 
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C is the average of the centroids of the regions, an estimate of the center 
of the mark; 

Vd is the vector from C to Cc, 
Wi is the angle of Vq ; 

c&. is the orientation of the major axis of the i th region; 
*, is the difference between the I* angle and the i th orientation; 
J 2 is the first performance measure of the regions analysis method; 
Ai is the area of the i th region, i € {1 • • • N/2} ; 
i* = i + (AT/2), it is the index of the region opposed to the i* region; 
Mi is the major axis length of the i th region; and 
rm is the minor axis length of the i th region. 
Eauations (62)-(66) compute a performance measure based on the fact that symmetrically 
~»Si - the mark 320 shown in Fig HA are e q ually disto^ y tr— 
and rotations when the artwork is far from the camera (i.e., in the far ^ field), and 
lXZLo«e* when the artwork is in the near field. Additionally the fact that the 
In" are elongated with the major axis oriented toward the center is used. Equation 
S d Tnle/the centroid of the combined regions from the cen.oids of the^ 
egions In Eqn (65) the direction from the center to the center of each region is computed 
aTd compared with the direction of the major axis. The performance measure, J 2 s 
colZ based on the differences between opposed spokes in relation to the mean f 
el CpertT Note that the algorithm of Eqns (62) - (66) operates without a single 
^ed parameter. The regions analysis method is also found to give the center of the 
mark to sub-pixel accuracy in the form of C . 

Thresholding A possible liability of the regions analysis method is that it requires 
"In of aluLosity threshold in order to produce a binary ^ -* "J* 
t With the need to determine a threshold, it might appear ^^"^ 
1 image would influence detection of a marl, even with the use of essentrally closed-path 

"t* ? „, threshold is determined for each scan. By gathering the luminosities, as for 
Fi g !6B and setting the threshold to the mean of that data, the threshold clouds 
Xo *e pixels under the closed path - which are guaranteed to fall on a detected mark 
. ie not influenced by uncontrolled regions in the trnoge. 

Performing region labeling and analysis across the image for «d. ~ ^ 
prohibitively expensive in some applications. But if the image ,s threshold at several 

of scanning operations can be performed with only a few labelling operates. In one 
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single labeled image. 

X3 Intersecting Edges Analysis 

these nn— at a = ^ ^ ~ points „' ^ on lh e W edges 
citified into N groups by pairing edges such as a and 9 , 6 and ft. *. » g 

the groups of edge points and j G {UN, W ) is an maex 

Each set of edge points defines a best-fit line, which may be given as: 



mean (xj) 
mean (yj) 



(67) 



(68) 



i. e J* to a scalar parameter describing position along the line, ft, 6 R 2 is one 

where a, € R is a scalar pn defining 
w __~ t v, P i; nP eiven as the means of the x s and y, values 01 tne eu B H 

ft "a v^r describing .ne siope of tne Une. Tne vaiues ft and „ are 
obtained, for example, solving for each group: 



1 X\ 
1 Xi 



n* = (*f*i)" 1 *? 



y2 



|i = 90° - atan (TU (2)) 



(69) 



(70) 



where the xj and yj are the 



X and Y coordinates of image points within a group of edge 
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the offset and slope of the i th line, and & € R 1 [degrees] 
le. Equation (69) minimizes the error measured along 
it is desirable to minimize the error measured along an 



the Y axis. For greatest precision : 

axis perpendicular to the line. This is accomplished by the refinement: 



while 5ii > £ 3 do 



'A (2) 'P 2 (2) 



cos sin (|i) 

—sin (li) cos 



'Pi(l) 'ft(l) 



'Pi(i) 'ft(i) 



'Pi(i) J p 2 (i) •••] 



= + 



(71) 
(72) 

(73) 
(74) 



where «P, (1) and <P, (2) refer to the first and second elements of the <P, e ^i^ vector 
respectively; e s provides a stopping condition and is a^small number, such a, 10 , 
/>• in Eon (67) is given by: £i = [ cos (&) sin (&) j • 

* ^linimum distance * bleen a point (5 and the I* best-fit line is given by: 



(75) 



c* = Af (<? "i) / 1 Ad 2 

d i = \C-(Cl i + a i iii)\ 

The best-fit intersection of a collection of lines, C, is the point which minimizes the 
sum of squared distances, E< * between <7 and each of the lines. The sum of squared 
distances is given by: 



(76) 



n d =[(5(l) C(2) «i «2 •••] 



N 
0 

-Ai (i) 
-Mi) 
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0 -fa (1) 

N -fa (2) 

-Ai(2) Ai(i) 2 + Ai(2) 2 
-A 2 (2) o 
; o 



EiLi-2a(i) 

E^-2fii(2) 
2fii (l)Ai (l) + 2fii(2)Ai (2) 
2n 2 (l)A2(l) + 2fi 2 (2)A2(2) 



-A 2 (i) 

-M2) 
0 

mi) 2 +M2) 2 

o 

T 



0 
0 



(77) 



(78) 



where Q d is the sum of squared distances to be minimized, C (1) , fi, (1) fa (D refer to 
the X-axis element of these vectors, and C (2) , ft, (2) fa (2) refer to the Y-axis element 
of these vectors; II d € is a vector of the parameters of the solution comprising the 

X- and y-axis values of C and the parameters a, for each of the N lines, and matrix 
A d e R(»W+» and row vector B d e i?<" +2 > are composed of the parameters of the N 

best-fit lines. _ N « 

Equation (76) may be derived by expanding Eqn (75) in the expression Q d - £ i=1 4 • 

Equation (76) may be solved for C by: 

n d = - (2A d ) _1 ( 79 ) 



C = 



n d (i) 
n d (2) 



0(1 




'^(3)' 






n d (4) 









(80) 



The degree to which the lines defined by the groups of edge points intersect at a common 
point is defined in terms of two error measures 

£l : the degree to which points on opposite edges of opposing regions fail to lie on a line, 

given by 

ei(i) = E^( 2 ) 



(81) 



with l Pj as given in Eqns (71)-(72), evaluated for the i th line. 
£2 : the degree to which the N lines connecting points on opposite edges of opposing 
regions fail to intersect at a common point, given by: 

,d? ( 82 ) 

t 



£a (i) = 23 d i 
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with dk as given in Eqn (75). 
In summary, the algorithm is: 

edges a and g, etc.; 

f,™d for the N groups of points using Eqns (67)-(74), and the 

2. AT best-fit lines are found tor the jy group v determined, 
errorbywhichthesepointsfailtolieonthecorrespondmgbest.fitlme.determx 

giving £i {%) for the i th group of points; 

i intPrsection of the N best-fit lines is 

3. The centroid C which is most nearly at the intersection 

determined using Eqns (75)- (80); 

4 . The distance between each of the best-fit lines and the centroid C is determined, 
giving £ 2 (i) for the i th best-fit line; 

5. The performance is computed according to: 



J 3 = l/I>l« + £ ^)} 



(83) 



K4 Combining Detection Approaches 

limited to this example. 

. Thresholding and labeling the image « 10 logarithmic^ spaced threshold, 

between the minimnm and maximum luminosity. 
. Essentia clced-path scanning and region analysis, as described in section K2., 
giving performance measure J 2 of Eqn (66). 

In some cases there are no false-positives at this point. 



38. 
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. Further refinement by evaluating the phase rotation giving Jx of Eqn (53). 
• Merging the performance measures 

J T = Jj J 2 Jz (84) 



9 
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L: Position and Orientation Estimation 



LI. Introduction 

Revive position and orientation in three dimensions (3D) between a scene reference 
equate system and a camera coordinate system (i.e., camera extenor onen^on) 
comprises 6 parameters: S positions { X, Y and Z } and 3 orientals { p, ch, roll and 
Taw} Some conventional standard machine vision techniques can accurate* measure 3 
of these variables, X-position, Y-position and roll-angle. 

The remaining three variables (the two ont-of-plane fit angles p tch and yaw ^and 
the distance between camera and object, or are difficult to esfmate at all -g 
o— tl machine vision technic and virtually impossible to estimate accnrat* 
ITventh variable, camera principal distance, depends on the zoom and focns of the 
canTa and may be known if the camera is a calibrated metric camera, or more l,k ly 
nZwn if the camera is a conventional photographic camera. This vanable „ also 
difficult to estimate using conventional machine vision techmques. 

Ll.l. Near and far field 
Using orientation dependent reflectors (ODRs), pitch and yaw can be measured 
A clin^ one embodiment, in the far-field (when the ODRs are far from the camera) 
tte me—nt of pitch and yaw is not coupled to estimation of Z- P os,fon or prmcpa. 

" —to another embodiment, in the near-neU estimates of prt* ya. 
Z-position and principle distance are coupled and can be ^f^^Z 
increases the complexity of the algorithm, bu, yields thebenefit o full ***?££Z 
(DOF) estimation of position and orientation, with estimation of prmcrpal d,stance 

added benefit. 

L2. Coordinate frames and transformations 
L2A. Basics 

The following material was introduced above in Sections B and C of the Description of 
thP Related Art, and is treated in greater detail here. 

Z mage metrology analysis, it is helpful to describe points in space w.th respe* 
to many coordinate systems or frames (such as reference or camera coordura.es). As as 
dis31ove hi connection with Figs 1 and 2, a coordinate system or frame general* 
"s three orthogonal axes (X, Y and Z>. fn general the - 'J^f ~ 

be described with respect to frame S by specifying its posrfon along each f th ~ axes, 
for example = [3.0, 0.8, l.»f . We may say that point B ,s descnbed » frame S, 
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in "the 5 frame," or equivalent!* "in 5 coordinates." For example, describing the position 
of point B with respect to (w.r.t.) the reference frame, we may write "point B m the 
reference frame is ...» or equivalent^ "point B in reference coordinates is . 

As illustrated in Fig 2, the point A is shown with respect to the camera frame c and 
is given the notation <P A . The same point in the reference frame r is given the notatum 

'^The position of a frame (i.e., coordinate system) relative to another includes both 
rotation and translation, as illustrated in Fig 2. Term «Po, refers to the locatxon of 
the origin of frame r expressed in frame c. A point A might be determined m camera 
coordinates (frame c) from the same point expressed in the reference frame (frame r) 

using / R c\ 

e P A = e r R r PA+- e Po r 1 > 

Where °R € * 3 * 3 expresses the rotation from reference to camera coordinates, and <P 0r 
is the position of the origin of the reference coordinate frame expressed in the camera 
frame. Eqn (85) can be simplified using the homogeneous coordinate transformation from 
frame c to frame r, which is given by: 



r cT = 



r Po c 



where 

IR 6 R 3x3 is the rotation matrix from the camera to reference frame, 
r P Dc € R 3 is the center of the camera frame in reference coordinates. 

A homogeneous transformation from the reference frame to the camera frame is then 

given by: 

"POr 



° T R 



(86) 



Where C T R = r c R T and c Po r = ~ C r R rp o*- # . 

Using the homogeneous transformation, a point A might be determined m camera 
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coordinates from the same point expressed in the reference frame using 

C P A = c r T t Pa (87) 

To use the homogeneous transformation, the position vectors are augmented by one^ 

For example <P A = f 3.0 0.8 1.2 f becomes <P A = 3.0 0.8 1.2 1.0 ] , with 1.0 
For example, r A [o j fl4 x4 The not ation 

adjoined to the end. This corresponds to r R € R while r i « 
C p A is used in either case, as it is always clear by adjoining or removing the fourth 
element is required (or third element for a homogeneous transform in 2 dimensions). In 
general, if the operation involves a homogeneous transform, the additional element must 
be adjoined, otherwise it is removed. 

L2.2. Rotations: 

Two coordinate frames axe related to each other by a rotation and translation as 
illustrated in Fig 2. Generally, the rotation matrix from a frame B to a frame A is 

given by: 



A X B a Yb a Ze 



(88) 



where *X B is the unit X vector of the B frame expressed in the A frame, and likewise 
for A Y B and *Z B . There are many ways to represent rotations in three dimensions, the 
most general being a 3x3 rotation matrix, such as *R . A rotation may also be described 
by three angles, such as pitch ( 7 ), roll (fl and yaw (a), which are also illustrated m Fig 

2 ' To visualize pitch, roll and yaw rotations, two notions should be kept in mind: 1) 
what is rotating; and 2) in what order the rotations occur. For example, 
embodiment, a reference target is considered as moving in the camera frame or coordmate 
Item ThJs, if the reference target was at the origin of the reference frame 74 shown 
in Fig 2 a +10° pitch rotation 68 (counter-clockwise) would move the Y-axis to the left 
and theZ-axis downward. Mathematically, rotation matrices do not commute, and so 

-Rroll flyaw #pitch ^ ^ aw i W» R ^ a 

Physically, if we pitch and then yaw, we come to a position different from that obtained 
from yawing and then pitching. An important feature of the pitch-yaw-roll sequence used 
here is that the roll is last, and so the roll angle is that directly measured in the image. 

According to one embodiment, the angles 7 , fi and a give the rotation of the 
reference target in the camera frame (i.e., the three orientation parameters of the exterior 
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orientation). The rotation matrix from reference frame to camera frame, & , is given 
by: 



IR = -R18O ^roll Ryaw Rpitch 
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(89) 



-CpC a -CpS^ + SpC^ -CpS^-SpS^ 
SpC a SpS Q S^ + CpC y SpSaC^-CpS^ 

s a -c a s, -OA 

where Cp indicates a cosine function of the angle fi , Sp indicates a sine function of the 
tle L the diagonal array reflects a 180" rotation of the camera frame about lt s Y- 
2 so "at the Z-L of the camera is pointed toward the reference target (in the sense 
^e the Z-axis of the reference frame, see Rotate, no— irna^me below). 
The rotation from the camera frame to the reference frame is given by: 



;h = c r R T = i^uch fl£ w ^oll ^80 
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•c j + v, 0 thpn vaw then roll of the reference target. 

Orientation is specified as the pitch, then yaw, meu 



(90) 
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L2.3. Connection to photogrammetric notation 
An alternative notation sometimes found in the photogrammetric literature is: 

Roll k (rather than p) 
Yaw tp (rather than a) 
Pitch w (rather than 7) 

The order of the rotations is commonly like that for ? JL 
L24- Frames 

Fot image oology anaiysis — g .o one embodiment there are severai coordinate 
frames (e.g., haviBg *. or three dimensions) that are cohered. 

1. Reference frame r P A 

T be referer.ce frame is aligned with the scene, centered in ^™TS«Z 

between the scene and reference frames. 
2 Measurement frame m Pyi 

£ fromthe" reference frame to the measurement frame may he gwen by: 



nrp __ 



(91) 



where 



* ™R = 



Ca 5 ^7S 



(92) 



A and <v, are arbitrary known yaw, roll and pitch rotatzons between the 
where a 5 , ft, and 7 s are axbi y ^ rf ^ 

reference and measurement frames, and F Qr .is tne po 
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reference frame in measurement coordinates. As shown in Fig 5, for example, the 
Ltor -Pa, could be established by selecting a point at which measurement plane 

23 meets the reference plane 21. 

b the parties example of Fig 5, the measurement plane 23 is ~ 
Plane 21 by a -90° yaw rotation. The information that the yaw rotator, * 90 * 
liable for built spaces with surfaces at 90= ang.es, and specialized information 
may be arable in other circumstances. The sign of the rotation must be con-stout 
with the 'right-hand rule,' and can be determined from the .mage. 
When there is a -90° yaw rotation, equation (91) gives 



0 0-1 
0 10 
10 0 
0 0 0 



X POr 



(93) 



3. ODR frame d >Pa 

Coordinate frame of the ODR. It may be rotated with respect to the reference 



frame, so that 



-s Pi 



0 



0 
0 

1 



(94) 



where Pj is the roll rotation angle of the j» ODR in the reference frame. The 
Action vector of the longitudinal (i.e., primary) axis of the ODR region . g.ven 

^-^[10 Of < 95 > 
In the examples of Figs 8 and 10B , the roll angles Pj of the ODRs is 0 or 90 degrees 
^ the reference frame. However, it should be appreciated that Pj may be an 
arbitrary roll angle. 

4. Camera frame c Pa 

Attached to the camera origin (i.e., noda, point of the lens) the is ou^f 

the camera, toward the scene. There is a 180 yaw rotafon between the reference 
td camera frames, so that the Z-a*is of the reference frame is pomtmg generally 
"ward the camera, and the Z-a*is of the camera frame is pointing generally toward 
the reference target. 
5. Image plane (pixel) coordinates % P a 
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Location of a point a (i.e., a projection of an object point A) in the image plane of 
the camera, *P a € R? ■ 

6. Normalized image coordinates n P a 
Described in section L3., below. 

7. Link Frame L P A 

The Z-axis of the link frame is aligned with the camera bearing vector 78 (Fig 
9), which connects the reference and camera frames. It is used in interpretation 
reference target reference objects to determine the exterior orientation of the 
camera. 

The origin of the link frame is coincident with the origin of the reference frame: 

'^ = [0 o of 
The camera origin lies along the Z-axis of the link frame: 

T Po e = IR [ 0 0 Zcam ] T 

where Zeun is the distance from the reference frame origin to the camera origin. 
8. Scene Frame 3 Pa 

The reference target is presumed to be lying flat in the plane of the scene, but there 
may be a rotation (-y axis on the reference target may not be vertically down in 
the scene). This roll angle (about the z axis in reference target coordinates) is given 
by roll angle /J 4 : 

C/u -Sp A 0 

C* 0 (96) 
0 0 1 



° r R = 



L2.5. Angle Sets 

From the foregoing, it should be appreciated that according to one embodiment, an image 
processing method may be described in terms of five sets of orientation angles: 

1. Orientation of the reference target in the camera frame: T c R(rr, P, <*)> 0- e -> the three 
orientation parameters of exterior orientation); 

2. Orientation of the link frame in the reference frame: r L R (72, aa), (i-e., camera 
bearing angels); 

3. Orientation of the camera in the link frame: \R (73, £3, "3) ; 
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4. Roll of the reference target (i.e., the reference frame) in the scene (arising with a 
reference target, the Y-axis of which is not precisely vertical): ° r R (ft); and 

5. Orientation of the measurement frame in the reference frame, ^Jlfoj, ft, « 5 ) 
(typically a 90 degree yaw rotation for built spaces.) 



L3. Camera Model 

By introducing normalized image coordinates, camera model properties (interior 
orientation) are separated from camera and reference target geometry (exterior 
orientation). Normalized image coordinates is illustrated in Fig 41. A point P A 51 
in the scene 20 is imaged where a ray 80 from the point passing through camera origin 
66 intersects the imaging plane 24 of the camera 22, which is at point «P„ 51'. 

Introducing the normalized image plane 24' at Z c = 1 [meter] in camera coordinates, 
the ray 80 from T P A intersects the normalized image plane at the point "P„ 51'. To 
determine «P 0 from knowledge of the camera and scene, 'P A is expressed in camera 
coordinates: 

C P A = C T T r P A 

where C P A = [ C X A C Y A C Z A f • 

Normalizing so that the Z-component of the ray 80 in camera coordinates is equal to 

1 meter, r „ 

r a — 



c z A 



C Y A / C Z A 
1 



(97) 



Eqn (97) is a vector form of the collinearity equations discussed in Section C of the 
Description of the Related Art. 

Locations on the image plane 24, such as the image coordinates -P., are determined 
by image processing. The normalized image coordinates «P tt are derived from >P a by: 



stepl: P a = ?T i P a 



(98) 



step 2 



1 a — 



P„(3) 



where P„ G R 3 is an intermediate variable, and *„T is given by: 



— dk x 
0 
0 



0 

— d k 
0 



■y V0 



(99) 
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Where 

i T e ^3x3 is a homogeneous transform for the mapping from the two- 
dimensional (2-D) normalized image coordinates to the 2-D image coordinates. 
d is the principle distance 84 of the camera, [meters]; 

kx is a scale factor along the X axis of the image plane 24, [pixels/meter] for 
a digital camera; 

ky is a scale factor along the Y axis of the image plane 24, [pixels/meter] for 
a digital camera; 

x 0 and y 0 are the X and Y coordinates in the image coordinate system of 
the principle point where the optical axis actually intersects the image plane, 
[pixels] for a digital camera. 

For digital cameras, k x and k y are typically accurately known from the manufacturers 
specifications. Principle point values x 0 and y 0 vary between cameras and over tune, 
and so must be calibrated for each camera. The principal distance, d depends on zoom, 
if present, and focus adjustment, and may need to be estimated for each image. The 
parameters of J.T are commonly referred to as the "interior orientation" parameters of 
the camera. 

L3.1. Image distortion and camera calibration 
The central projection model of Fig 1 is an idealization. Practical lens systems will 
introduce radial lens distortion, or other types of distortion, such as tangential (i e, 
centering) distortion or film deformation for analog cameras (see, for example, the 
Atkinson text, Ch 2.2 or Ch 6). 

As opposed to the transformations between coordinate frames, for example r l , 
described in connection with Fig 1, image distortion is treated by mapping within one 
coordinate frame. Locations of points of interest in image coordinates are measured by 
image processing, for example by detecting a fiducial mark, as described in Section K. 
These measured locations are then mapped (i.e., translated) to locations where the points 
of interest would be located in a distortion-free image. 

A general form for correction for image distortion may be written: 

«J^ = /.(U,«P.) - ( 10 °) 

where f c is an inverse model of the image distortion process, U is a vector of distortion 
model parameters, and, for the purposes of this section, <P; is the distortion-free location 
of a point of interest in the image. The mathematical form for /„ (U, •) depends on the 
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distortion being modeled, and the values of the paramet^ depend on the detaUs ofthe 
camera and lens. Determining values for parameters U is part of the process of came™ 
calibration, and must generally be done empirically. A model for radral lens d.stortron 

may, for example, be written: 



r a = \/{Xa - So)* + (Va ~ Vo) 

5r a = K 1 rl + K 2 rl + K 3 r 7 a 



(101) 
(102) 



6x a = 5r a ^ ; 6y a = 5r a % 



i P„* = i P„ + <5 i P« = 







Sx a 


Va , 


+ 





(103) 



(104) 



where mapping /. W •) is given by E q ns (101)-(104), -P. « [ x V. ] « ^ measured 
location of point of interest a, for example at 51' in Fig 1, U = [ K* K ,] T £ the 
vector of parameters, determined as a part of camera calibration, and 5*P a is the offset m 
imag e location of point of interest a introduced by radial lens distortion. Other distortion 
models can be characterized in a similar manner, with appropriate functions replacing 
Eqns (101)-(104) and appropriate model parameters in parameter vector U. 

Radial lens distortion, in particular, may be significant for commercial digital cameras. 
In many cases a single distortion model parameter, will be sufficient. The parameter 
may be determined by analyzing a calibration image in which there are sufficient control 

points (i.e., points with known spatial relation) spanning a sufficient region of the image. 

Distortion model parameters are most often estimated by a least-squares fitting process 

(see, for example, Atkinson, Ch 2 and 6). 

The distortion model of Eon (100) is distinct from the mathematical form, mos 

commonly used in the field of Programmetry (e.g., Atkinson, Ch 2 and Ch 6), but 

has the advantage that the process of mapping from actual-image to normahzed unage 

coordinates can be written in a compact form: 



i p — n T 



fdurPa) 

1 



(105) 



where »P„ is the distortion-corrected location of point of interest a in normalized image 
coordinates "T = *T _1 e R 3x3 is a homogeneous transform matrix, [ f c (U, *P„J 1 J 
the augmented vector needed for the homogeneous transform representation, and 
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function f c {U, ) includes the non-linearities introduced by distortion. Alternatively, 
Eqn (105) can be written 

n P a =iT( i P a ) ( 106 > 
where the parentheses indicate that ?T(-) is a possibly non-linear mapping combining 
the nonlinear mapping of U {U, •) and homogeneous transform ?T. 

Using the notation of Eqn (100), the general mapping of Eqn (9) of Section A may 

be written: / 1 \ „„n 

n-/ c ^- n- 1 cp>l (3) A ) 

and the general mapping of Eqn (10) of Section A may be written: 

i P _ r i ( u i t — ^-rr e T T t Pa\ ( 108 ) 

fa-U c p A (3) r * J 

where *P a is the location of the point of interest measured in the image (e.g., at 51' in 
image 24 in Fig 1), f~ l {U, 'Pa) is the forward model of the image distortion process 
(e.g., the inverse of Eqns, (101)-(104)) and * n T and are homogeneous transformation 
matrices. 

L4. The image metrology problem, finding t Pa given l P a 

Position 'P A can be found from a position in the image 'P.. This is not simply 
a transformation, since the image is 2 dimensional and *P A expresses a point m 3- 
dimensional space. According to one embodiment, an additional constraint comes from 
assuming that r P A lies in the plane of the reference target. Inverting Eqn (98) 

n p nrp i p* 

To discover where the vector »P„ intersects the reference plane, the vector is rotated into 
reference coordinates and scaled so that the Z-coordinate is equal to T P 0c (3) 

V. = IR n P* W 

r Pi _ _ r PQc(3) r j + rp (110) 

where ' J„ is an intermediate result expressing the vector from the camera center to "P« 
in reference coordinates, and 'Po c (3) and 'P (3) refer to the third (or Z-axis) elements 
of each vector, respectively; and where 'R includes the three orientation parameters of 
the exterior orientation, and T P Qc includes the three position parameters of the exterior 

orientation. 



171 



The method of Eqns (109)-(110) is essentially unchanged for measurement in any 
coordinate frame with known spatial relationship to the reference frame. For example, if 
there is a measurement frame m (e.g., shown at 57 in Fig 5) and ?R and m P Qr described 
in connection with Eqn (91) are known, then Eqns (109)-(110) become: 



m J a = ?R n P° ( 1U ) 



mp„ _ m PorXV mj + mp (112) 
r A — m j 0 ( 3 ) J a -T * O c 



where m P 0e = m Po T + ? R rp o e - 

The foregoing material in this Section is essentially a more detailed treatment of the 
discussion in Section G of the Description of the Related Art, in connection with Eqn (11). 
Eqns (111) and (112) provide a "total" solution that may also involve a transformation 
from a reference plane to a measurement plane, as discussed above in connection with 
Fig 5. 

L5. Detailed discussion of Exemplary Image Processing Methods 

According to one embodiment, an image metrology method first determines an initial 
estimate of at least some camera calibration information. For example, the method 
may determine an initial estimate of camera exterior orientation based on assumed, 
estimated, or known interior orientation parameters (e.g., from camera manufacturer). 
Based on these initial estimates of camera calibration information, least-squares iterative 
algorithms subsequently may be employed to refine the estimates. 

L5.1. An Exemplary Initial Estimation Method 
One example of an initial estimation method is described below in connection with the 
reference target artwork shown in Figs 8 or 10B. In general, this initial estimation method 
assumes reasonable estimation or knowledge of camera interior orientation parameters, 
detailed knowledge of the reference target artwork (i.e., reference information), and 
involves automatically detecting the reference target in the image, fitting the image of 
the reference target to the artwork model, detecting orientation dependent radiation 
from the ODRs of the reference target, calculating camera bearing angles from the ODR 
radiation, calculating a camera position and orientation in the link frame based on the 
camera bearing angles and the target reference information, and finally calculating the 
camera exterior orientation in the reference frame. 
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L5.1.1. An Exemplary Reference Target Artwork Model (i.e., Exemplary 
Reference Information) 

1. Fiducial marks are described by their respective centers in the reference frame. 

2. ODRs are described by: 

(a) Center in the reference frame T Po Dj 

(b) ODR half length and half width, (length2 , width2) 

(c) Roll rotation from the reference frame to the ODR frame, 



Cos pj Sin pj 
—Sin pj Cos pj 



where pj is the roll rotation angle of the j th ODR. 

L5.1.2. Solving for the reference target geometry 
Determining the reference target geometry in the image with fiducial marks (RFIDs) 
requires matching reference target RFIDs to image RFIDs. This is done by 

1. Finding RFIDs in the image (e.g., see Section K); 

2. Determining a matching order of the image RFIDs to the reference target RFIDs; 

3. Determining a center of the pattern of RFIDs; 

4. Least squares solution of an approximate coordinate transformation from the 
reference frame to the camera frame. 

L5.1.3. Finding RFID order 
The Nrios robust fiducial marks (RFIDs) contained in the reference target artwork are 
detected and located in the image by image processing. From the reference information, 
the N FIDs fiducial locations in the artwork are known. There is no order in the detection 
process so before the artwork can be matched to the image, it is necessary to match the 
RFIDs so that r 0 corresponds to 'Or,, where '0 Fj e R 2 is the location of the center 
of the of the j» RFID in the reference frame, <0 Fj e R 2 is the location of the center 
of the j th RFID detected in the image, where j € {1..AW- To facilitate matching 
the RFIDs, the artwork should be designed so that the RFIDs form a convex pattern. 
If robustness to large roll rotations is desired (see step 3, below) the pattern of RFIDs 



- 173- 

should be substantially asymmetric, or a unique RFID should be identifiable in some 
other way, such as by size or number of regions, color, etc. 

An RFID pattern that contains 4 RFIDs is shown in Fig 40. The RFID order is 
determined in a process of three steps. 

Step 1- Find a point in the interior of the RFID pattern and sort the angles ^ to each of 
the N FIDs RFIDs. An interior point of the RFID pattern in each of the reference and 
image frames is found by averaging the N FIDs locations in the respective frames: 

r O F = mean ( t Ofj) 
•Of = mean (^f,) 

The means of the RFID locations, 'O f and *O f provide points on the interior of 

the fiducial patterns in the respective frames. 
Step 2: In each of the reference and image frames, the RFIDs are uniquely ordered by 
measuring the angle between the X-axis of the corresponding coordinate frame 
and a line between the interior point and each RFID, such as in Pig 40, and 
sorting, these angles from greatest to least. This will produce an ordered list of the 
RFIDs in each of the reference and image frames, in correspondence except for a 
possible permutation that may be introduced by roll rotation. If the is little or no 
roll rotation between the reference and image frames, sequential matching of the 
uniquely ordered RFIDs in the two frames provides the needed correspondence. 

Step 3 Significant roll rotations between the reference and image frames, arising with 
either a rotation of the camera relative to the scene, 0 in Eqn (92), or a rotation 
of the artwork in the scene, ft in Eqn (96), can be accommodate by exploiting 
either a unique attribute of at least one of the RFIDs or by exploiting substantial 
asymmetry in the pattern of RFIDs. The ordered list of RFIDs in the image (or 
reference) frame can be permuted and the two lists can be tested for the goodness 
of the correspondence. 

L5.14- Finding the ODRs in the image 
Three or more RFIDs are sufficient to determine an approximate 2-D transformation from 
reference coordinates to image coordinates. 
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where *O f € R 3 is the center of an RFID in image coordinates augmented for use with a 
homogeneous transformation, «T 2 € R™ is the approximate 2-D transformation between 
essentially 2-D artwork and the 2-D image; and '0 Fj € R 3 is the X and Y coordinates 
of the center of the RFID in reference coordinates corresponding to 'Of,, augmented for 
use with a homogeneous transformation. 

The approximate 2-D transformation is used to locate the ODRs in the image so 
that the orientation dependent radiation can be analyzed. The 2-D transformation is so 
identified because it contains no information about depth. It is an exact geometric model 
for flat artwork and in the limit z cam -> oo. When the reference artwork is flat, and the 
distance between camera and reference artwork, Zcam , is sufficiently large. Writing 



'Of. = 



fitting of: 


*x Fl 




*yFi 








. % HFn vid , . 









a b c 




T X Fj 


^Fj 


= 'T 2 '0 Fj = 


d e f 




T VFi 


1 




0 0 1 




1 


, e, and / of transformation matrix ;T 2 can 


T x Fl 


r y Fl i 


0 




0 


0 


o o 


r x Fl 




T VFx 




0 




0 


0 


o o 


rXF * F ,D. 





(113) 



0 
1 

0 

1 



a 
b 
c 
d 
e 
/ 



Once *T 2 is determined, the image region corresponding to each of the ODRs may be 
determined by applying « r T 2 to reference information specifying the location of each ODR 
in the reference target artwork. In particular, the corners of each ODR in the image may 
be identified by knowing * r T 2 and the reference information. 

L5.1.5. Detecting ODR radiation 
Based on the fiducial marks, two-dimensional image regions are determined for each ODR 
(i e ODR radiation pattern), and the luminosity in the two-dimensional image region is 
projected onto the primary axis of the ODR region and accumulated. The accumulation 
challenge is to map the two-dimensional region of pixels onto the primary axis of the 
ODR in a way that preserves detection of the phase of the radiation pattern. This 
mapping is sensitive because aliasing effects may translate to phase error. Accumulation 
of luminosity is accomplished for each ODR by: 

1. Defining a number iV bins (j) of bins along the primary axis of the j th ODR; 
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2. For each pixel within the image region of the jtn ODR, determining fc, the index of 
the bin into which the center of the pixel falls, 

3. For each bin the sum and weighted sum of pixels falling into the bin are accumulated 
so that the mean and first moment can be computed: 



(a) The mean luminosity in bin fc of the f h ODR is given: 



(114) 



i=l 



where JV^fc) is the number of pixels falling into bin fc; A (i) is the measured 
luminosity of the i th image pixel, and L is the mean luminosity; 
(b) The of the center of luminosity (the first moment) is given: 



*A(*) = £ Mi)*p(i)/M*) 



(115) 



where ^(fc) € R 2 is the first moment of luminosity in bin fc, ODR j, and 
i P (i) e B? is the image location of the center of pixel *. 

L5.1.6. Determining camera bearing angles a 2 and 7 2 from ODR rotation 
angles 0j 

The Z-axis of the link frame connects the origin of the reference frame center with the 
origin of the camera frame, as shown at 78 in Fig 9. The pitch and yaw of the Imk frame 
referred to as camera bearing angles (as described in connection with Fig 9), are denved 
from the respective ODR rotation angles. The camera bearing angles are a 2 (yaw or 
azimuth) and 72 (pitch or elevation). There is no roll angle, because the camera beanng 
connects two points, independent of roll. 

Rotation from the link frame to the reference frame is given by: 



r R = 



C a2 


0 






' 1 


0 


0 


0 


1 


0 




0 






~~ Sat 


0 


Ca2 




0 




C 72 



Ca 2 

0 



^72 

Ca2 ^72 



-s. 



'72 
C*2^72 



The link frame azimuth and elevation angles a 2 and 72 are determined from the ODRs 
of the reference target. Given 0, the rotation angle measured by the j ODR, is 
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given by the first element of the rotated bearing angles: 

0,. = ?'#(!,:) 



72 



Where the notation ?'*(!, :) refers to the first row of the matrix. Accordingly, pitch and 

yaw are determined from 9j by: 



72 






-1 


' 6i' 


Oil 








02 . 



(the matrix pseudo-inverse would be use if more than two ODR regions were measured). 
The camera bearing vector is given by: 





0 




C72 Sct2 


r Po c = IR 


0 








^cam 







Expressing the bearing vector in the ODR frame gives 
D iPo = D jR rp Qc + D iPor = 



C72 ^"2 ^72 



72° Pj 



a 2 ^72 



Zc^+^POr ( 116 ) 



The measured rotation angle, B s is related to the bearing vector by: 

0J . = arctan [ Dj p oA3) ) 



(117) 



When the center of the reference frame is on the Y-axis of the ODR frame it follows that 
Dj p 0r (1) = 0j an d *»P 0r (3) = 0. Accordingly, Eqns (116) and (117) can be combined to 

PVe " C ^C Pi S at - S 12 S Pj _ fnn ^ (118) 



Ca2 ^*72 



With the ODR angles, measured and the reference information known, there are two 
unknowns in Eqn (118): 72 and a 2 . Bringing these terms out, we can write: 



a 



Pi 



'P2 



-5 
-s 



PI 



P2 









' tan(^i) 




' hi ' 




tan (0 2 ) 




h 2 





(119) 
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where n a <? 

All = — 



and =5, 



/l2 = 



72 



Solving Eqn (119) for and h 2 allows finding a 2 and 7 , If there are many ODRs, Eqn 
(119) lends itself to a least squares solution. The restriction used w.th Eqn (118) tha 
o iPo (i) = o, and »iP*(3) = 0, can be relaxed. If z cem » |[ D >Po r (1) 'ftv (3) j| 
Eqn (118) will be a valid approximation and the values determined for a 2 and 72 close 
to the true values. 

L5.1.7. Calculating camera position and orientation in the link frame; 
derivation of ° L R and c Po r 
Using projective coordinates, one may write: 

c Pa(3) n Pa= c PA (120) 



Where 

*P A is the 3-D coordinates of a fiducial mark in the camera coordinate system 
(unknown); 

»P a is the normalized image coordinates of the image point 'P. of a fiducial 
mark (known from the image); 

cp A (3) is the Z-axis coordinate of the fiducial mark P A in the camera frame 
(unknown). 

Using Eqn (120) and the transformation of reference to camera coordinates, one may 

fiDd: *P A = *P A (3) n P a = [ e L * *R t Pa + C Po] < 121 ) 

Where <Po, is the reference frame origin in the camera frame (unknown), and also 
represents the camera bearing vector (Fig 9). 

Rotation *R is known from the ODRs, and point 'P x is known from the reference 
information, and so <<P A (and likewise L P B ) can be computed from: 

L P A =?R t Pa+ L Po r ( note : Lp Or = 0 bv definition) 

Using at least 2 fiducial marks appearing in the image of the reference target at reference- 
frame locations 'P A and 'P B known from the reference information, one may write: 
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d A n P a = IR L PA + C POr 
d B = IR + CP Or 

where d A = -P^S) and d B = c Pb(3) [meters]. These two equations may be viewed as 
"modified" collinearity equations. 

Subtracting these two equations gives: 

dA * Pm - dB *n=Vi (^p a - l p b ) ( 122 ) 

The image point corresponding origin (center) of the reference frame, <P 0r , is determined 
for example using a fiducial mark at 'P*. an intersection of lines connecting fiducial 
marks, or transformation «T a . Point -Pa,, the normalized image point corresponding 
to 'Pa,, establishes the ray going from the camera center to the reference target center, 
along which °Z L lies: 

°Z h = - n PoJ \\ n Por\\ 
Rotation %R may be written 



C L R = [ C X L c Yl c Zl } 



where X L etc. are the unit vectors of the link frame axes. The rotation matrix is given 
as: 



C rR = 



Cf, 3 C a3 Sp 3 C a3 s as 

C fii S as S-y a + C^Sp 3 -^ 3 S QS 5 7S + C^C 73 -CA 
- C y3 Cp 3 S a3 Cp 3 S^ 3 + C^ 3 S a3 Sp 3 Ca.Cn 

And so a 3 and 73 may be found from: 

a 3 = 180 - sin -1 ( c Z L (l)) 

where 180° is added because of the 180" yaw between the camera frame and the link 
frame. The range of sin" 1 is -90"..- 90°. The pitch rotation from camera frame to link is 

^ 73 = atan2 (- *Z L (2) /C Q3 , °Z L (3) /C a3 ) (124) 



(123) 



Writing 
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= L P A - L P B 



Eqn (122) may be written: 

d A n P a - d B n P b = b x + S Ps d 2 ) + b y (Cfrdv - 5/j.di) + b z d 3 



(125) 



Where 





Ca 3 




0 




»5a3 






^2 = 














5 73 




C03 C73 



dl and & are seen to represent the first two columns of S fl with the & terms factored 
out. Eqn (125) can be rearranged as: 



d A ( n Pa ~ n Pb) + dAB n Pb = C h ex + Sp 3 e 2 + b z d 3 



(126) 



with 



d AB = d A -d B ; e 1 = b x d l + b y d 2 and = Ma - V*i 

System of equations (126) provides four equations in four unknowns, 3 equations from 
the three spatial dimensions, and the nonlinear constraint: 

cfc + al-i < 127) 

The unknowns are: { d A ^ (Iab* Cfa Sp 3 }- 
This system of equations can be solved by: 

1. Setting up the linear system of three equations in four unknowns: 

Q = [(np a -np 6 ), "ft, -el, -e2] 



5 = 



d A = g Pa&) 

dAB 
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b z d 3 = QB 

2. The matrix Q G fl 3x4 . The solution comprises a contribution from the row space 
of Q and a contribution from the null space of Q. The row space contribution is 
given by: 

B r = Q~ R b z d 3 = Q T inv (Q Q t ) b z d 3 (128) 

3. The contribution from the null space can be determined be satisfying constraint 
(127): 

B = B r + ^N Q (I 29 ) 
Where N Q is the null space of Q, and ip G R 1 is to be determined. 

4. Solve for ip : 

B(4) = B r (4) + ^7V Q (4) 
B(3) 2 + B(4) 2 -l = 0 (130) 

Which gives: 

qitf) 2 + q2ip + Qs = 0 
qx = N Q (3) 2 + N Q (4) 2 
q 2 = 2 (B r (3) N Q (3) + B r (4) N Q (4)) 
93 = J3 r (3) 2 + B r (4) 2 -l 
5. There are two solutions to the quadratic equation 



* = 2q~i 

The correct branch is the one which gives a positive value for d A = B (1) = C P A (3). 
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With the solution of Eqn (129) values for { d M d AB , Cp 3 , S Ps } are determined 
and IR can be found. Vector c P 0h is approximately given (exactly given if r P A = 
[ 0 0 0 ] T ) as: 

0 



c Po h = IR 



0 

d A 



(131) 



Steps 1 through 5, combined with Eqns (123) and (124) provide a means to estimate 
the camera position and orientation in link coordinates. As described in SectionL5.1.6., 
interpretation of the ODR information permit estimation of the orientation of the link 
frame in reference coordinates. Combined, the position and orientation of the camera in 
reference coordinates can be estimated. 

L5.1.8. Completing the initial exterior orientation estimation (i.e., resection) 
The collinearity equations for resection are expressed in Eqn (10) as: 

i P a = i c T{ e r T( r P A )) 



where, from Eqn (91) 



:r 



0 ' : 1 

Using the link frame as an intermediate frame as discussed above: 

c r R = c L R h r R 

where ?R was determined in Section L5.1.6. using information from at least two ODRs, 
and C L R and c P 0r = c Po L were determined in Section L5.1.7. using information from at 
least two fiducial marks. From %R the angles a, p, 7 can be determined. 

L5.1.9. Other exemplary initial estimation methods 
Alternatively to the method outlined in Sections L5.1.6. and L5.1.7., estimates for the 
exterior orientation parameters may be obtained by: 

1. Estimating the pitch and yaw from the cumulative phase rotation signal obtained 
from a robust fiducial mark, as described in Section K, Eqn (59); 
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2 Estimating the roll directly from the angle between a vector between two fiducial 
marks in the reference artwork and a vector in image coordinates between the 
corresponding images of the two fiducial marks; 

3. Estimating the target distance (z cam ) using the near-field effect of the ODR 

discussed in Appendix A; 

4. Estimating parameters «J>* (1) / «Po,(3) and *P 0r (2) / °Po r (3) from the image 
coordinates of the origin of the reference frame (obtained using a fiducial mark at 
the origin, the intersection of lines connecting fiducial marks, or transform matrix 

5. Combining estimates of * cam , 'Pa, 0)/ e Por (3) and <Po r (2) l*Po T (3) to estimate 
c Po r - 

Other methods to obtain an initial estimate of exterior orientation may also be used; in 
one aspect, the only requirement is that the initial estimate be sufficiently close to the 
true solution so that a least-squares iteration converges. 

L5.2. Estimation Refinements; Full Camera Calibration 
A general model form is given by: 

v = F{u,c) ; e = v-v ( 132 ) 

where v € R™ is a vector of m measured data (e.g., comprising the centers of the 
fiducial marks and the luminosity analysis of the ODR regions); v e R™ is a vector 
of m data predicted using the reference information and camera calibration data; F (•) is 
a function modeling the measured data based on the reference information and camera 
calibration data. The values for reference information and camera calibration parameters 
are partitioned between u e the vector of n parameters to be determined and c 
a vector of constant parameters. The several parameters of u + c may be partitioned 
many different ways between « (to be estimated) and c (constant, taken to be known). 
For example, if parameters of the artwork are precisely known, the reference information 
would be represented in c. If the camera is well calibrated, interior orientation and image 
distortion parameters would be placed in c and only exterior orientation parameters would 
be placed in «. It is commonly the case with non-metric cameras that principle distance, 
d is not known and would be included in vector u. For camera calibration, additional 
interior orientation and image distortion parameters would be placed in u. In general, 
the greater the number of parameters in «, the more information must be present in the 
data vector v for an accurate estimation. 
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Vector u may be estimated by the Newton-Raphson iteration described bek™ This 
is one embodiment of the generalized functional model described m Section H and with 
Eqn (14), and is described with somewhat modified notation. 

u 0 : inital estimate of the scaled parameters 



fori = 0... (Ni-1) 



Vi = F(ui, c) 



6i=Vi-V 



(133) 



5iii = 



tt m = iii + Siii 



where 



Ni is the number of iterations of the Newton-Raphson method; 

v e R m is the measured data; 

d e R m is the estimation residual at the i th step; 

S e R nxn is a matrix scaling the parameters to improve the conditioning of 
the matrix inverse 



S = 



si 



5 2 



u 0 = S' 1 u 0 are the scaled initial parameters; 
W € R™*™ is a matrix weighting the data; 
. The iteration of Eqn (133) is run until the size of the parameter update is less than 

a stop threshold, |«ti<| < StopThreshold. This determines N ti 
. The partition of model parameters between u and c may vary, it is not necessary 

to update all parameters all of the time. 
. Scaling is implemented according to « = Si where u is the vector of the 
parameters. The matrix inversion step in Eqn (133) may be poorly condition if 
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the parameters span a large range of numerical values, scaling is used so that the 
elements of u axe of approximately the same size. Often, the « are chosen to have 
magnitude comparable to a typical value for the corresponding «*. 
There are several possible coordinate frames in which e, can be computed, including image 
coordinates, normalized-image coordinates and target coordinates. Image coordinates 
are used, because the data are directly expressed here without reference to an> . model 
parameter. Using image coordinates requires that the derivatives of equation (133) all be 
computed w.r.t. image coordinate variables. 

To carry out the iteration of Eon (133) the data predicted by the model, must be 
computed, as well as the derivatives of the data in the parameters, fc)^. Computation 
of these quantities, as well as determination of 5 and W is discussed in the next three 
sections. 



L5.2.1. Computing Vi 
The data predicted on the basis of reference information and camera calibration are given 
by 

'Of, (1) 
*O n (2) 



U (i) 

Li (iV bins (l)) 



Vi = F{ui-i, c) = 



FID positions in image 
RGR Luminosity 



(134) 



L Nd (1) 
L Nd {Nuns {N D )) 

where *O r (1) is the predicted X-coordinate of the fiducial mark, and likewise ^ (2) 
is the predicted Y-coordinate; N D is the number of ODR regions, and where the predicted 
luminosity in the fc* bin of the 3 th ODR region is written L^k), j € {1.JVd} and 
k € {L.Nuns (J)}, the range for k indicates that a distinct number of bins may be used 
for the accumulation of luminosity for each ODR region (see section L5.1.5.). 

In the reference artwork examples of Figs 8 and 10B, there are 8 data corresponding 
to the measured X and Y positions of the 4 fiducial marks, and 344 data corresponding 
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to luminosity as a function of position in a total of four ODR regions (in Fig 8, the 
ODRs shown at 122A and 122B each comprise two regions, with the regions arranged 
by choice of grating frequencies to realize differential mode sensing, in Fig 10B there are 
four ODRs, each with one region and arranged to realize differential mode sensing). 

The predicted fiducial centers are computed using the equations (97) and (98). The 
luminosity values are predicted using Eqn (48): 

"£j(k)= ?T(*Pj(*)) 

D J a = c f R n Pj (k) 

Lj (k) = ao (j) + a, (j) cos {v (tPj(k))) 

where i Pj(k) € R 2 is the first moment of illumination in the k th bin of the j th ODR; 
fpj(k) E R? is the corresponding point projected to the front face of the ODR (using 
camera calibration parameters in u and c); Lj (fc) is the value of luminance predicted by 
the model at front-face point fpj(k) (using parameters a Q (j) and a x (j)). 

L5.2.2. Determining the data derivatives with respect to the model parameters 

For the purposes of discussion, estimation of- the exterior orientation, principle distance 
and ODR parameters {u 0 , a 0 , ax} is considered in this section, giving u € R 7+3N °. For 
the artwork of Figs 8 and 10B, with two ODR regions per ODR, N D = 4, which gives 19 
parameters, or u e R 19 . The order of the model parameters in the u vector is: 

u= [ 7 p a c P t d ... n Duo i O0 i ai ... " DOo "x> ai ] T (135) 

Where C P% G .R 3 represents the reference artwork position (reference frame origin) 
in camera coordinates. Alternative embodiments might additionally include the three 
additional interior orientation parameters and the parameters of an image distortion 
model. 

Determining the derivative of the fiducial mark positions w.r.t. model parameters 

The image-coordinate locations of the fiducial marks (RFIDs) are computed from their 
known locations in the artwork (i.e., reference information) using the coordinate transform 
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given in Eqn (108) where Z" 1 (• •) and %T and *„T depend upon the exterior and interior 
orientation and camera calibration parameters but do not depend on the ODR region 
parameters. 

Computation of the derivative requires which depends on r P A and is given by: 



d c P A _ 
(in 



A ' dP A ' da. 
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(136) 



where 



d c P* e rMi+Wd) . 

and where ^ 6 # 3x3 is the element-wise derivative of the rotation matrix w.r.t. 7. 



Prom Eqn (92), one finds 



d Hpitch 



dZR DDR d/t pi* 
— iM80 stroll *Vaw a 7 



-1 


0 


0 




" Cp 




0 ' 




" c a 


0 


s a ' 




0 


1 


0 








0 




0 


1 


0 




0 


0 


-1 




0 


0 


1 






0 


c a 





10 0 

0 -S 7 -C 7 
0 C 7 -5 7 



(137) 



and likewise for the other three rotation matrix derivatives. 

Starting with Eqns (97) and (98) the derivatives in pixel coordinates are given by: 

~dt ■> ~ aK * \ °Pa(3) <*« c Pa(3) du ' 

d<P a ,» \_ ( 1 d°P A (2,:) °P*(2)d°PA(3,:)\ 

(.2, :J - -aKy \ cp A ( 3 ) du ^PaW du I 



(138) 



ilfa. (■ 7) = - 
du 



A/ a* 

0 k 



0 



c .Pa(3) 



where sub-arrays are identified using MATLAB notation: A (1, :) refers to the 1 st row of 
A, B (:, 7) refers to the 7 th column of B; and if C is a 3-vector, C (1 : 2) refers to elements 
1 through 2 of vector C. When r P A in Eqn (136) is the position in reference coordinates 
corresponding to r O Fj , the position of the j th RFID, then ^ is the derivative of the 
position in image coordinates of the j th RFID w.r.t. parameters u. 
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L5.2.3. Determining the derivative of orientation dependent radiation, Lj (k) 
w.r.t. model parameters 
The computation of the derivatives of Lj {k) proceeds by: 

1. The known pixel coordinates of the center of luminosity of each bin are transformed 
to a point on the ODR, 'Pj(fc) — > f Pj(k) 

2. The derivative of the transformation, d'Pj{k)/d , is computed: 



d r Pp c 

du 



d^T ~ Or > d/3 ° r ' da 
du fc 





' o " 






c Po r , r c R, 


0 




(139) 
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^ = Ja (I) [- Dj„(3) du ^ "JaW du J V 

D J„(3) du ^ du 

16 {1,2} (I 42 ) 

3. The point tpj(k) is projected onto the ODR region longitudinal axis 

'p 3 (fc)»= r x Dj { r x T Di {jpm- rp o))+ Tp o* 

where *X Dj is the unit vector along the longitudinal axis of the ODR region in 
reference coordinates, and r P Qc is the reference coordinates center of the ODR 
region. 

4. The derivative ^ is calculated at /£,(*)«. The derivative follows from Eqns 
(J21) and (J22). 

5. The derivative of the back grating shift w.r.t. the parameters is computed: 

^, d6» X ( r Yn ryT ££31*1 _ (143) 
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6. The component of *j£ lying along the ODR longitudinal axis is considered 

d6 Db x Rl * 7 d5 Db x = rx r d5^x (U4) 
du du du 

7. The derivative of the Moire pattern (i.e., triangle waveform) phase at a point in the 
image w.r.t. the parameters is given by: 

^ e RW») ; 

j S = [[360 (/ / -A)^^S fii + 360^] [0 •-• 1 ••• 0]jl46) 

where the vector [o - l --o]e i? lxN ' reflects the contributions of the u 0 
parameters to the derivative. 
8. Finally, the derivative the Moire pattern luminance at a point in the image w.r.t. 
the parameters is given by: 



[ [-^sin^)^] [0 ••• 1, cos(i/) ••• 0]] 



(146) 



where the first term has dimension lx(7 + iV c ) and includes derivatives due to 
the extended exterior orientation parameters and the u 0 , and the second term has 
dimension lx2N D and includes derivatives w.r.t. the parameters oq and ^ (see Eqn 
(J25)). 

9. The derivative of the data w.r.t. the parameters is given by: 



dv _ 
du 



e R (2N Fm +N,)x{7+3N D ) (147) 



Where JV FID is the number of fiducial marks, N, is the total number of luminosity 
readings from the N D ODR regions, and the zero terms, [0] €E reflects 
the fact that the fiducial locations do not depend upon the ODR region parameters. 

L5.3. Determining the weighting and scaling matrices 
The weighting matrix W and the scaling matrix S play an important role in determining 
the accuracy of the estimates, the behavior of the iteration and the conditioning of the 
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matrix inversion. The matrices W € W *» , . 

typically diagonal, which can be used to improve the efficiency of the evaluate of Eqn 
(133). The elements of W provide a weight on each of the data points. These weights 
are used to: 

. Shut off consideration of the ODR data during the first phase of fitting, while the 
fiducial marks are being fit; 

• Control the relative weight placed on the fiducial marks; 

. Weight the ODR luminosity data, (see Eqn (114)), according to the number of 

pixels landing in the bin; 

• Window the ODR luminosity data. 

The elements of 5 are set according to the anticipated range of variation of each variable. 
For example, <Po, (3) may be several meters, while d will usually be a fractionof a meter; 
therefore S (6, 6) takes a larger value than 5 (7, 7). 5 (6, 6) corresponds to Pa (3), and 
5(7,7) corresponds to d (see Eqn (135)). The diagonal elements of W and S are non- 

negative. 
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M Summary of Exemplary Implementations 

„ should be appreciated tha, a variety of image metrology methods and apparatus 
.cording to the present invention, including those parity described in detai. above, can 
be imp.emen.ed in numerous ways, as the invention is no, limited to any particular manner of 
imputation. For examp.e, image metro.ogy methods and apparati* according ,0 vanous 
embodiments of me invention may be indented using dedicated hardware design* to 
perform any one or more of a variety of functions described herein, and/or using one or more 
computers or processors (e.g., the processor 36 shown in Fig. 6, the client workstation 
processors 44 and/or the image metroiogy server 36A shown in Fig. 7, etc.) mat are 
programmed using microcode (i.e., software) to perform any one or more of the vane* of 

functions described herein. 

,„ particular, it should be appreciated tha, the various image metroiogy methods 
outlined herein, inc.uding the de.ai.ed mathematical analyses outiined in Sections J, K and L 
of the Detailed Description, for example, may be codec as software tha, is execute on a 
processor ma, empioys any one of a variety of operating system, Additionahy, such software 
may be written using any of a number of suitab.e programming .anguages and/or tools, 
inc.uding, bu, no, .imited ,o, <he C-programming ianguage, MATLAB™, MathCAD™, and 
me like, and also may be compiled as executable machine language code. 

,» mis respec,, it should be appreciated that one embodiment of the invention .s 
direct ,o a compter readable medium (or multiple computer readable media) (e.g., a 
computer memory, one or more floppy discs, compact discs, optica, discs, magnetic .pes, 
etc.) encoded with one or more computer programs that, when executed on one or more 
computers or other processors, perform methods that implement the various embodiments of 
me invention discussed above. The comp»,er readab.e medium or media can be transportable, 
such mat me program or programs stored thereon can be .oaded onto one or more different 
computer syaems to imp.emen. various aspect of the present invention as discussed above. 
- .t should be underst^d ma, the term "computer program" is used herein in a generic sense to 
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refer to any type of computer code that can be employed to program a computer or other 
processor to implement various aspects of the present invention as discussed above. 
Additionally, it should be appreciated that according to one-aspect of this embodiment, one or 
more computer programs tha, when executed perform methods of the present invention need 
no, reside on a single computer or processor, but may be distributed in a modutar fashron 
amongst a number of different computers or processors to implement various aspects of the 
present invention. 

Having thus described several illustrative embodiments of the present invention, 
various alterations, modifications, and improvements will readily occur to those skilled in the 
art Such alterations, modifications, and improvements are intended to be within the spin, and 
scope of me invention. Accordingly, the foregoing description is by way of example only and 

is not intended as limiting. 
What is claimed is: 



